Cameras capture everything that happens inside a physical space. But without a structured memory layer, that footage is a passive record — not a queryable source of intelligence. SpatialMem is a memory-centric AI system that changes this. Starting from casually captured egocentric RGB video, it reconstructs metrically-scaled indoor environments, organises them into a hierarchical spatial memory, and makes every object and location searchable using natural language — with no specialist sensors required.
What Problem Does SpatialMem Solve?
Existing AI systems that work with camera footage can classify and describe what they see in a single frame. What they cannot do is answer questions that require memory across time and space: Where is the object I saw earlier? How far is the exit from the anomaly I detected? What was visible from that position?
SpatialMem addresses this gap directly. It unifies 3D geometry, semantics, and language into a single, queryable representation — giving AI agents the ability to reason about physical environments the same way a person who has walked through a space might recall and navigate it.
How SpatialMem Builds Spatial Memory
SpatialMem processes casually captured egocentric RGB video in a hierarchical two-layer pipeline.
Layer 1 — Structural Scaffold: The system detects permanent architectural features — walls, doors, windows — and uses them as metrically-anchored reference points. These structural anchors form a stable first-layer scaffold that persists even as objects in the scene change or move.
Layer 2 — Object Memory: The system populates the scaffold with open-vocabulary object nodes. Each node links evidence patches, visual embeddings, and two-layer textual descriptions to precise 3D coordinates. This design enables compact storage and fast retrieval without requiring pre-defined object categories — the system handles descriptions it has never been explicitly trained on.
Spatial Reasoning and Language-Guided Retrieval
The hierarchical memory structure enables interpretable spatial reasoning that existing video AI systems cannot perform. SpatialMem supports queries over:
- Distance — how far between two objects or locations
- Direction — the spatial relationship between positions in the scene
- Visibility — what is and is not visible from a given vantage point
These capabilities support downstream tasks including language-guided object retrieval and offline navigation-style guidance over a pre-built memory — without requiring a live camera feed or any specialist depth-sensing hardware.
Benchmark Results
SpatialMem was evaluated across one public Replica scene and two real-world egocentric indoor scenes, testing performance under increasing levels of scene clutter and occlusion.
- Layout reasoning: Stable anchor-description-level navigation completion across all evaluated scenes
- Hierarchical retrieval: Strong accuracy maintained as scenes become more complex
- Two-layer description memory: Ablation confirms this improves path-level language grounding over single-layer alternatives
- Scale robustness: Moderate metric scale perturbation causes only limited performance degradation, confirming practical resilience outside controlled lab conditions
Key Insights
- No specialist hardware required: SpatialMem operates from standard egocentric RGB video. Depth sensors, LiDAR, and dedicated mapping equipment are not needed.
- Open-vocabulary language support: The system handles object descriptions it has not been explicitly trained on, making it practical in real-world environments where the unexpected is routine.
- Two-layer memory structure outperforms flat alternatives: Separating structural anchors from object nodes improves both retrieval accuracy and spatial grounding under clutter.
- Scales to real-world conditions: Performance remains stable across tested scenes despite occlusion, variable lighting, and clutter levels typical of operational environments.
Why This Matters for Physical AI
SpatialMem positions camera footage as an active, queryable memory layer for physical space — not a passive recording. For environments where AI agents and robots need to operate across long time horizons, this is a foundational capability. An agent that can ask "where did I last see that object?" and receive a spatially grounded, navigable answer is a qualitatively different kind of system than one confined to understanding the present frame.
This research represents a key building block in the visual memory infrastructure that Memories.ai is developing for enterprise and operational environments.
Conclusion
SpatialMem demonstrates that metrically-scaled, language-queryable spatial memory can be constructed from ordinary camera footage, without specialist sensors, and maintained accurately under the clutter and occlusion conditions typical of real indoor environments. It offers an efficient and extensible memory interface for spatially grounded long-horizon video understanding — one that can be connected to AI agents via standard retrieval interfaces.
Read more
O-MARC: Omni Memory-Augmented Compression Distillation for Efficient Video Understanding
Omnimodal AI models that understand both video and audio together represent the current frontier of video understanding, but processing long, joint token sequences makes inference slow and memory-intensive. O-MARC introduces a new public benchmark that demands genuine audio-visual reasoning, and a compression distillation framework that makes omnimodal inference 34.6% faster and 34.7% more memory-efficient — while improving accuracy over full token inference.
MARC: Memory-Augmented RL Token Compression for Efficient Video Understanding
Large Language Models (LLMs) are rapidly evolving into powerful multimodal systems (VLMs), but when you switch from a single image to a long, high-frame-rate **video**, the computational cost explodes. This immense computational overhead—due to the sheer number of visual tokens—creates significant latency and memory bottlenecks, making it nearly impossible to deploy high-performing VLMs in real-time, resource-constrained applications like autonomous driving, surveillance, and real-time video Q\&A.
UGC-VideoCaptioner: An Omni UGC Video Detail Caption Model and New Benchmarks
User-generated content (UGC) videos—like TikToks—combine fast-paced visuals, music, and speech, requiring deep multimodal understanding. Existing benchmarks are mostly visual-only, neglecting the crucial role of audio. To address this, researchers introduce **UGC-VideoCap**—a new benchmark for detailed audio-visual captioning—and **UGC-VideoCaptioner-3B**, a compact model distilled from Gemini-2.5 Flash using supervised fine-tuning (SFT) and GRPO reinforcement learning.