Memories.ai Brings Real-World Data and Post-Training Expertise for NVIDIA Cosmos

At Memories.ai, we help robotics and industrial teams build physical AI for their specific tasks, combining real-world video data with post-training expertise. A robot learning to fold laundry or a model watching a production line needs to recognize the hands, objects and steps involved—and know when something goes wrong. Our video data lake provides the examples needed to teach those capabilities.

We build on NVIDIA Cosmos, using its physical-world understanding as the foundation. We use Cosmos to interpret video entering our data lake, then curate that data to post-train Cosmos for customers’ robotics and industrial applications. This loop turns real-world video into models tailored to the tasks our customers need to solve.

The loop: real-world video in, post-trained Cosmos models out

Our system runs as one loop:

• Capture. Egocentric video from wearable cameras, including 5K+ Luci Pins recording worldwide, plus dedicated head and wrist camera setups.
• Index. Our captioning model, fine-tuned from Cosmos, describes each clip. Captions and embeddings are keyed to timestamps, so teams can search for moments instead of files.
• Annotate. Cosmos-based annotation models label what the hands are doing, step by step, alongside 3D hand pose and action boundaries.
• Post-train. We post-train Cosmos on the curated data for each customer's task.
• Deploy. The post-trained model runs on NVIDIA hardware, from data center GPUs to NVIDIA Jetson at the edge.
• Improve. A data agent finds where the deployed model fails and pulls the right video from the lake. A training agent retrains. Our scientists stay in the loop to define what "good" means for each task.

Every turn of the loop makes Cosmos better at the real world, and makes our data lake better at teaching it.

Three ways we post-train Cosmos

1. Understand Egocentric Video for Data Lake Indexing

Indexing millions of hours of egocentric video is itself a physical AI problem. The model has to understand hands, tools and objects seen from the wearer's point of view, which general-purpose video models handle poorly.

We fine-tuned our captioning model from Cosmos, and we are moving our indexing models onto Cosmos. In our testing, Cosmos is stronger on egocentric and physical-world video than the general-purpose vision language models we used before.

2. Annotate Egocentric Video for Robotics

Robotics teams do not just need video. They need video labeled the way a robot learns: which hand, which object, which step, and where the step started and ended.

We post-train Cosmos to annotate egocentric manipulation video, and we help robotics companies post-train Cosmos for their own annotation needs. Each clip in our data lake carries five time-aligned layers: action boundaries, action class, dense captions, task labels and 3D hand pose. On Meta's HOT3D benchmark, our hand pose wrist error is 10.4 mm, compared with 16.8 mm for WiLoR, 33.6 mm for HaMeR and 51.9 mm for MediaPipe.

In November we will open-source our egocentric hand-action captioning model, built on the Cosmos family, on Hugging Face, so any robotics team can start from it.

3. Industrial Inspection on Jetson

The same loop works outside robotics. For an industrial video inspection task, we post-trained Cosmos 3 Edge with LoRA on our curated data and deployed it end to end on NVIDIA:

• 8.7% → 93.4% accuracy after post-training, while training only about a third of one percent of the weights
• 8.6 points better than a 4B open model, with 39% fewer parameters and less than half the training time on RTX PRO 6000
• On a NVIDIA Jetson Orin NX, with DeepStream and NVIDIA TensorRT: 0.51 seconds per stream against a one second budget, 27.6 W, and a cold start cut from two minutes to six seconds

Same foundation, same loop, a different slice of the physical world.

Why post-training, and why now

Many teams want Cosmos to work zero-shot, and for many tasks it does. But the tasks that matter most in physical AI, the ones with real hands, real objects and real consequences, are exactly where post-training on real-world video makes the difference. Very few teams do that work today, because the hard part is not the training run. It is having the right video, checked, indexed and labeled well enough to teach from.

That is the gap Memories.ai fills. NVIDIA built the foundation and the tools: Cosmos models, the Cosmos post-training framework, and the inference stack from DeepStream and TensorRT to Jetson. We bring the real-world video and the loop that turns it into better models.

What's next

We are extending the loop in three directions:

1. More of the Cosmos framework. Moving more of our post-training onto the Cosmos framework's SFT and upcoming RL recipes.
2. Hand pose inside Cosmos. Training 3D hand pose estimation directly into our Cosmos models, so one model can index, caption and track hands in a single pass.
3. More robotics teams. Opening the data lake to more robotics and world-model teams who want to post-train Cosmos on egocentric video of real tasks.

Memories.ai builds video data infrastructure to accelerate physical intelligence. To post-train Cosmos on your own video, or to work with our egocentric data, visit memories.ai.