RAG Anything Files

What's new

Audio & video modalities (#292)

Audio and video are now first-class optional modalities:

• AudioModalProcessor — local ASR via faster-whisper with timestamped segments; long recordings are windowed into ordered chunks (token-budget bounded), no content dropped.
• VideoModalProcessor — dual-channel: SceneDetect scene boundaries + OpenCV keyframes + VLM visual description + audio-track transcription, merged by timestamp; long videos spill into multiple ordered chunks.
• Opt-in via enable_audio_processing / enable_video_processing (env ENABLE_AUDIO_PROCESSING / ENABLE_VIDEO_PROCESSING), default off. Install extras: pip install raganything[audio] / raganything[video] (video also needs ffmpeg on PATH). With deps missing or flags off, media items fall back to the generic processor.

Media files as direct inputs (#354)

process_document_complete("meeting.mp3") and folder batches now work: audio/video files short-circuit the parser dispatch into a single media content item (no document parser involved) and flow through the normal multimodal path. Items carry a streaming SHA-256 of the file, so replacing a recording at the same path re-ingests it as new content.

Markdown image extraction (#352)

Markdown image references (![alt](path)) that resolve to readable local files become image blocks shaped exactly like MinerU's and flow through the same multimodal pipeline as images extracted from PDFs — lossless originals, no PDF re-encode. URLs and missing files stay literal text. Fenced code blocks are never interpreted (this also stops # inside fences from becoming headings). The old PDF round trip never actually carried images (ReportLab rendered the syntax as literal text), so this implements the original intent for the first time.

Fixes

• Test-session sys.modules corruption from test_modalprocessors_config.py (#353) — stub teardown now restores real modules; unrelated tests no longer fail depending on execution order.

Docs

• CJK audio guidance: use WHISPER_MODEL=medium or larger with WHISPER_LANGUAGE set — measured on the same Mandarin clip, base garbles proper nouns, medium recovers them (#355).

Validation

Beyond the unit suites (411 tests, macOS/py3.10 + Linux/py3.11), this release was exercised end-to-end on a clean Linux host with real dependencies and live LLM/VLM/embeddings:

• audio formats wav/mp3/flac (two-speaker English meeting) and m4a/ogg-opus (Chinese narration), via both file-path and content-list entry
• video containers mp4/webm/mkv (4-slide lecture with narration): on-screen slide text and spoken facts both recovered
• 9.3-minute narration transcribed and windowed into multiple ordered chunks
• one document combining text + image + audio + video, answering a single query with facts from three modalities
• mixed folder batch (.md + .mp3 + .mp4) through process_folder_complete
• edge cases: silent video (no audio track), no-speech audio, force_multimodal_reprocess on media with no duplicate chunks
• real MinerU 3.4.5 parse of an image-embedded PDF (v2 content list) cross-checked against markdown image blocks
• whisper model ladder on Mandarin: base -> small -> medium quality progression verified; pipeline identical across runs

Full history: https://github.com/HKUDS/RAG-Anything/compare/v1.4.0...v1.4.1