MAGIC-Video: Structured Memory for Ultra-Long Agentic Video Reasoning

Jiazheng Li, Chi-Hao Wu, Yunze Liu, Kaize Ding, Jundong Li, Chuxu Zhang — University of Connecticut, Memories.ai, Northwestern University & University of Virginia
arXiv:2605.08271v1 [cs.CV], 8 May 2026

Overview

Understanding ultra-long videos such as egocentric recordings, live streams, or surveillance footage spanning days to weeks remains a challenge. For current multimodal LLMs, even with million-token context windows, frame budgets cover only tens of minutes of densely sampled video, and most evidence is discarded before inference begins. Memory-augmented and agentic approaches help with scale, but their retrieval remains fragmented across modalities and lacks long-range narrative summaries that span days or weeks. MAGIC-Video is a training-free framework built around a multimodal memory graph with an interleaved narrative chain: the graph unifies episodic, semantic, and visual content through six typed edges and supports cross-modal retrieval, while the chain distils long-horizon entity biographies and recurring activity events. At inference time, an agentic loop interleaves graph retrieval with narrative fact injection, covering both the modality and time dimensions of ultra-long video in a single retrieval pipeline. On EgoLifeQA, Ego-R1 and MM-Lifelong, MAGIC-Video outperforms strong general-purpose, long-video, and agentic baselines, with gains of 10.1, 7.4, and 5.9 points over the prior best agentic system on each benchmark.

Problem and Contributions

Video understanding has progressed from short clips of seconds to minutes, to medium-length content on the order of tens of minutes, and more recently to hour-scale videos. A further frontier is now emerging: ultra-long videos that span days, weeks, or even months. EgoLifeQA ships 7 days of continuous first-person footage, and MM-Lifelong extends to Day, Week, and Month subsets, with its Month split reaching 51 days of live-stream video. This regime is where many valuable real-world signals reside, such as personal memory assistants over recorded life-logs, agent activity logs accumulated over days, and long-term surveillance or livestream recordings. The questions they demand — how an entity evolves across days, which habits recur week to week, or when a plan was made and when it was carried out — are fundamentally temporal, entity-centric, and cross-modal.

Even frontier MLLMs with million-token contexts do not close the gap: a million-token budget covers only tens of minutes of densely sampled video, and at week scale any attempt to fit raw frames into the context window forces aggressive downsampling. An alternative is to build an external memory of the video offline and retrieve only the query-relevant evidence at inference time. Yet these memory-based systems, designed and evaluated on hour-scale video, fall short in two concrete ways.

Cross-modal retrieval remains fragmented. Representative systems retrieve each modality through an independent channel: WorldMM keeps three heterogeneous memory stores (episodic, semantic, visual) and relies on an adaptive agent to pick among them per query, while EGAgent couples a text-only entity graph with external visual-search tools. Even systems that place multimodal content in a single graph fall back on similarity search over individual nodes, so a query such as “who handed Jake the black marker during the discussion?”, which needs to jointly activate a visual clip and the semantic facts about the entities appearing in it, cannot be resolved through any single retrieval step.

Bottom-up aggregation dilutes fine-grained details. Representative systems build memory by summarising captions level by level and extracting entities and triples per local window: Vgent retrieves over chunked multi-scale captions, and VideoRAG indexes caption summaries at successively coarser granularities. As granularity coarsens, individual mentions of an entity and individual steps of an activity get abstracted away into generic window descriptions, and the very details the query relies on are gone. No top-down pass scans the whole video to identify recurring entities or multi-day activities and preserve their dated moments as coherent retrieval units.

Method

MAGIC-Video has two phases: an offline phase that processes each video once into a static memory artefact, and an online phase that, given a question, queries this artefact through a multi-round agentic loop.

Preprocessing

Preprocessing turns a raw long video into the five artefacts the rest of the pipeline consumes: multi-granularity episodic captions at 30 s / 3 min / 10 min / 1 h, named-entity annotations, consolidated semantic triples, per-clip visual embeddings, and a BM25 index over the captions. All five are produced by chaining ASR and a visual-caption model with an LLM-based merge, aggregation, and OpenIE pipeline.

Multimodal Memory Graph

The Multimodal Memory Graph is a heterogeneous graph that places all per-video evidence into a single retrieval substrate. It has four node types — Episode (multi-granularity captions), Visual-clip (per-clip visual embedding), Entity (NER mention, first-person canonicalised), and Semantic (consolidated triple) — and six typed edges: MENTIONED_IN, CO_CLIP, APPEARED_IN, HAS_PROPERTY, TEMPORAL_NEXT, and CONTAINS. This vocabulary gives the graph cross-modal shortcuts in a single traversal: textual seeds reach visual clips via MENTIONED_IN then CO_CLIP, and visual seeds reach semantic facts via APPEARED_IN then HAS_PROPERTY. Text embeddings are computed for episode and semantic nodes only; visual-clip and entity nodes have their embeddings zeroed and serve as PageRank bridges, with visual content arriving through a separate seed channel.

Narrative Memory Chain

While the graph supports local, caption-level retrieval, many life-long questions ask about what happens between retrieved captions — how an entity’s state changes, or how an activity recurs across weeks. The Narrative Memory Chain distils the captions offline into two complementary structures: topic chains that trace individual entities, and event chains that link related activity steps. Both are produced by feeding captions of progressively finer granularity to an LLM. A topic chain traces the lifecycle of a single entity and outputs a chronologically ordered list of timestamped facts; an event chain extracts per-day activities, identifies activities related across time (recurring routines, progressive project stages, or intermittent work) into dated chains, and enriches each step from the 30-second captions inside its time range.

Agentic retrieval

At inference time, MAGIC-Video exposes both structures to the reasoning backbone through a multi-round retrieval loop that alternates between search and answer. Given a search query, it runs one cross-modal Personalized PageRank pass over the time-filtered graph, seeded from three complementary channels — caption embedding, BM25, and visual — mixed into a single reset vector. Alongside the PageRank pass, the narrative chains are matched against the same query in two tiers: Tier 1 admits chains whose keyword entities appear in the query; Tier 2 admits chains by cosine similarity between the query and the chain’s content. Admitted facts are time-filtered and deduplicated against retrieved episodes. In each round the controller emits either search or answer; the loop terminates when the controller answers or after five search rounds.

Experimental Setup

Evaluation covers three question sets drawn from two ultra-long video corpora. EgoLifeQA and Ego-R1 Bench use the week-long egocentric life-logging recordings released with EgoLife; following prior work, evaluation uses the A1_JAKE subject, whose recording consists of 51.9 hours of continuous first-person footage over 7 days. EgoLifeQA provides 500 multiple-choice questions grouped into five subtasks — EntityLog, EventRecall, HabitInsight, RelationMap, and TaskMaster — while Ego-R1 Bench provides 50 questions split between a human-authored Manual set and a model-generated Gemini set. MM-Lifelong covers three temporal scales; the Month subset used here consists of 105.6 hours of live-stream video spanning 51 days, containing 623 open-ended questions across eleven categories.

For EgoLifeQA and Ego-R1 the metric is multiple-choice accuracy. MM-Lifelong is scored with an LLM-as-judge protocol, using GPT-5 as the judge for consistent comparison across methods. MAGIC-Video uses gpt-oss-120b for offline preprocessing and as the online retrieval controller, and Qwen3.5-Flash as the reasoning backbone. Node text embeddings are computed with Qwen3-Embedding-4B and visual-clip embeddings with VLM2Vec-Qwen2VL-7B, applied to 16 uniformly sampled frames per 30-second clip; audio is transcribed with Whisper-large-v3-turbo. The framework is training-free, and the same hyperparameters are used across all experiments wherever applicable.

Results

Consistent improvements across benchmarks

MAGIC-Video achieves the highest overall accuracy on every benchmark, despite running on a Qwen3.5-Flash reasoning backbone. On EgoLifeQA it reaches 67.6% average, surpassing the strongest prior system, EGAgent with a Gemini 2.5 Pro backbone, by 10.1 points; even with the much smaller backbone, the gap over the best single-shot frontier MLLM — Gemini 2.5 Pro at 3000 frames, scoring 46.8% — widens to +20.8. On Ego-R1 it reaches 64.7% average, surpassing the previous best system, WorldMM on a matched Qwen3.5-Flash backbone, by 7.4 points. On MM-Lifelong it reaches 24.5 under the GPT-5 judge, surpassing the best prior agentic system, ReMA-GPT-5, by 5.9 points, while the four 7–8B Long-Video MLLMs cluster at 6.3–9.1%.

The ranking is preserved when MM-Lifelong is scored with two alternative judges: MAGIC-Video scores 28.7 / 29.4 / 24.5 under Qwen3.5-Flash / GPT-5 Mini / GPT-5, versus at most 20.0 / 20.9 / 18.1 for the strongest General MLLM baseline, indicating that the observed advantage is not a judge-specific artefact.

Cross-modal and long-horizon subtasks benefit the most

On EgoLifeQA, the largest gains over WorldMM appear on EntityLog (+17.6), TaskMaster (+14.3), HabitInsight (+13.1), and EventRecall (+11.1) — the subtasks whose answers most directly require linking a text mention to a specific visual moment or spanning multiple days. RelationMap shows the smallest matched-backbone gap (+4.0), consistent with these queries being largely textually recoverable from the independent-retrieval baseline’s top-k episodes already. On MM-Lifelong the pattern is sharper: MAGIC-Video more than doubles the best Long Video MLLM on Causal Reasoning (29.4 vs. 7.9) and Event Recognition (36.1 vs. 15.7), and improves Temporal Reasoning from 13.9 to 27.1. Counting (7.4 vs. InternVideo2.5’s 12.6) and Hallucination Detection (15.9 vs. GPT-5 Mini’s 25.0) are the two categories where MAGIC-Video does not lead, because both reward dense single-pass frame coverage over multi-round retrieval.

Ablation

Stacking the two structured-memory components on top of a WorldMM-style independent-retrieval baseline isolates each contribution. The Multimodal Memory Graph adds +8.4 on EgoLifeQA and +4.0 on Ego-R1, and on MM-Lifelong restores a working retriever at 21.4 where the independent baseline runs out of host memory, its per-granularity HippoRAG indices exhausting host RAM. The Narrative Memory Chain adds another +3.2 / +3.4 / +3.1 to reach 67.6 / 64.7 / 24.5. Neither component alone reaches the full system, so the two are complementary rather than redundant.

Broken down by subtask, the graph’s +8.4-point EgoLifeQA lift concentrates on EntityLog (+12.0) and EventRecall (+11.1), both of which need the typed-edge path MENTIONED_IN then CO_CLIP to co-retrieve a text mention with its visual moment in one pass. The smaller RelationMap lift (+3.2) reflects a ceiling effect, as the IR baseline already reaches 62.4% there. Retrieved contexts are a non-trivial mix of Episode (63–74%), Semantic-triple (21–35%), and Visual-frame (2–10%) items across the three benchmarks, confirming that the retrieval pass activates every retrievable node type rather than collapsing to a text-only retriever. The Narrative Memory Chain fires on 46–54% of questions across the three benchmarks, with topic chains firing several times more often than event chains.

Efficiency

MAGIC-Video does not inflate the agentic search budget relative to the independent-retrieval baseline. On EgoLifeQA it uses slightly fewer retrieval rounds (3.33 vs. 3.49) and a lower rate of queries exhausting the five-round cap (29.4% vs. 32.2%), so pre-distilled chains let the agent reach an answer without spending more agentic turns. The per-question retrieval context is approximately 1.78× larger in text tokens (6,189 vs. 3,471) and includes approximately 3.75× more visual frames (51.0 vs. 13.6); both gaps reflect cross-modal retrieval delivering visual coverage on essentially every query, at 98% versus the baseline’s 15%.

Code and artifacts

The MAGIC-Video code is publicly available under the Apache-2.0 licence at github.com/lijiazheng0917/MAGIC-video, covering preprocessing, graph and chain construction, evaluation, and the baseline reproductions. Because the offline preprocessing steps call an LLM and are therefore non-deterministic, the authors also release the exact artifacts used in the paper — captions, OpenIE results, semantic triples, and topic and event chains for EgoLife A1_JAKE and MM-Lifelong — as a separate dataset at jiazhengli7/magic-video-artifacts. With those downloaded, reproduction requires only the deterministic GPU steps and evaluation. All GPU steps in the paper were run on a single NVIDIA A100 40 GB. MAGIC-Video builds on WorldMM, HippoRAG, and VLM2Vec.

Conclusion & Limitations

Conclusion. MAGIC-Video is a training-free structured-memory framework for ultra-long video reasoning that pairs a multimodal memory graph with a narrative memory chain. The graph unifies episodic, semantic, and visual evidence under six typed edges so that a single cross-modal Personalized PageRank pass propagates relevance across modality boundaries; the chain distils long-horizon entity biographies and recurring activity events that no single retrieved snippet can express. At inference time, an agentic loop combines per-round graph retrieval with narrative chain injection and feeds the merged context to a reasoning backbone. Across EgoLifeQA, Ego-R1, and MM-Lifelong, MAGIC-Video consistently outperforms strong general-purpose, long-video, RAG-based, and agentic baselines, with the largest gains on subtasks that require cross-modal or cross-day reasoning.

Limitations. The framework has three limitations, each suggesting a natural future direction. Caption-bounded recall: both retrieval and narrative distillation are upper-bounded by the raw captions, so an event not described in any caption cannot be surfaced by graph traversal or chain matching. Offline construction: both components are built once, offline, so streaming video — where captions, entities, and narratives must be updated as new footage arrives — is not yet supported. Higher per-question token consumption: cross-modal retrieval returns a multi-scale node mix and the narrative chain injects additional facts when a chain matches the query, which on EgoLifeQA raises the per-question retrieval context from approximately 3.5K to approximately 6.2K tokens; the round count itself does not increase, but the controller’s prompt grows accordingly.

Summary of arXiv:2605.08271v1. All figures and reported numbers are drawn directly from the source paper.