OmniRetriever: Any-to-Any Audio-Video-Text Retrieval via Fusion-as-Teacher Distillation

Authors: Yunze Liu, Chi-Hao Wu, Enmin Zhou, Junxiao Shen · Affiliations: Memories.ai Research

Cross-modal retrieval — finding the right content across different formats using a query in any format — is central to how AI agents interact with stored information. Existing systems handle video-text or audio-text retrieval in isolation. OmniRetriever is the first model to unify all three modalities — audio, video, and text — into a single retrieval space, enabling any modality to retrieve any other. Evaluated zero-shot across six benchmarks, OmniRetriever-7B surpasses closed-source Gemini Embedding 2 by 13.3 to 18.0 R@1 on audio retrieval, and reaches the contemporary specialist band on video-text tasks.

What Problem Does OmniRetriever Solve?

Unified multimodal embedding spaces — where different types of content are mapped to the same representational space so they can be compared and retrieved — have become standard infrastructure for AI retrieval and retrieval-augmented generation (RAG).

The problem is that existing audio-video-text (AVT) encoders are trained using pairwise objectives: they learn text-video alignment and audio-text alignment separately, in pairs. The combined (text, video, audio) signal that exists whenever all three modalities are present together is never used during training. This leaves substantial representational capacity on the table.

OmniRetriever closes this gap with a new training objective: fusion-as-teacher distillation.

Fusion-as-Teacher Distillation

The core innovation is a training method that leverages the joint three-modality signal as a teacher for the individual single-modality encoders.

When audio, video, and text are all present, OmniRetriever generates a fused embedding from all three. This fused embedding is then used as a stop-gradient teacher signal for each single-modal encoder during training — alongside a Tuple-InfoNCE term that supervises the fused embedding directly.

The result: single-modality encoders that, at inference time, produce embeddings shaped by the full three-way context they were trained against. Asking a text query returns results that reflect the audio and visual meaning of the content, not just textual similarity.

OmniRetriever-Bench: The First 12-Direction AVT Benchmark

Existing retrieval benchmarks evaluate single-modal directions only: text-to-video, or audio-to-text. No public benchmark covers the six dual-modal retrieval directions — queries that combine two modalities on one side (e.g., text-and-audio querying video; video-and-text querying audio).

OmniRetriever-Bench is the first benchmark to cover all 12 retrieval directions across audio, video, and text. It contains 3,782 held-out aligned triples, each scored against the same gallery across every direction. All 3,782 captions were reviewed and corrected by trained human annotators.

This benchmark enables the field to assess retrieval systems against the full range of real-world query types — including the dual-modal queries that are most relevant to practical AI agent deployments.

Benchmark Results

Audio-text retrieval (zero-shot) — Clotho and SoundDescs:

OmniRetriever-7B surpasses Gemini Embedding 2 by 13.3 to 18.0 R@1 on Clotho and SoundDescs in every direction, zero-shot — without task-specific fine-tuning.

On video-text retrieval benchmarks MSR-VTT and MSVD, OmniRetriever-7B reaches the contemporary zero-shot specialist band of open video-text encoders.

Key Insights

• Pairwise training leaves the joint signal unused: Standard audio-video-text encoders trained on pairwise objectives never see the full three-modality signal. Fusion-as-teacher distillation corrects this without changing the inference architecture.
• Single-modal queries benefit from three-modal training: At inference time, OmniRetriever accepts single-modal queries. The performance gains come entirely from the training methodology, not from requiring all three modalities at query time.
• Outperforms Gemini Embedding 2 zero-shot: The 13.3–18.0 R@1 advantage over Google's closed-source model on audio retrieval benchmarks demonstrates that the training approach — not scale alone — is the determining factor.
• 12-direction evaluation reveals real retrieval capability: OmniRetriever-Bench closes a significant measurement gap in the field. Prior single-direction benchmarks could not assess retrieval systems against the full range of query types that real AI agents require.

Why This Matters for AI Agents in Physical Environments

The practical implication of any-to-any retrieval is straightforward: AI agents operating in physical environments can query stored footage using whatever information is available. A question expressed in text can retrieve the relevant video clip. An audio signature — a specific alarm, a particular voice — can surface the matching footage. A visual description can find the matching audio event.

This is not a theoretical capability. It is the search layer that makes visual memory infrastructure genuinely queryable by AI agents connected to it — which is exactly what Memories.ai's CORE product is designed to provide.

Conclusion

OmniRetriever demonstrates that audio-video-text retrieval can be solved as a unified problem through fusion-as-teacher distillation, producing single-modality encoders that outperform closed-source competitors on zero-shot audio retrieval while matching specialists on video-text tasks. OmniRetriever-Bench provides the first comprehensive evaluation framework for the full 12-direction AVT retrieval space, establishing a new standard for measuring real-world retrieval capability.