In exploring how robots can continuously understand real-world environments, the research teams at Fudan University and Motion Brain have turned their attention to visual memory.
Intelligent agents operating in the physical world cannot afford to wait until a video ends before interpreting their surroundings. They must observe, remember, and update their understanding simultaneously—capturing ongoing actions and changes in object states while retaining information about events that occurred minutes earlier.
To address this challenge, FreshMem introduces a brain-inspired hybrid frequency-spatial memory framework, offering a new approach to streaming video understanding that balances recent visual details, long-term semantic information, and computational efficiency.
From “Watch Then Answer” to “Understand While Watching”
Many existing multimodal large language models process videos under the assumption that the entire video is available at once. While this offline paradigm works well for video retrieval and content analysis, it differs fundamentally from the real-world conditions faced by robots, smart glasses, and mobile intelligent agents.
In these environments, surroundings change continuously, future frames are not yet available, and the system must interpret and respond based solely on information observed up to the current moment.
This creates a fundamental trade-off. Retaining every frame leads to continuously increasing computational and memory costs. Discarding older frames, however, risks losing critical information about people, object states, and causal relationships between events. Common approaches such as uniform frame sampling, sliding windows, and simple compression can help control resource consumption, but they may also cause irreversible loss of important details or fragment the continuity of long-term events.
For embodied intelligence, this challenge is even more immediate. An intelligent agent must understand its surroundings online: newly appearing obstacles require detailed perception, while earlier interactions need to be consolidated into stable event memories.
The key is not to store every frame, but to enable the model to determine what deserves high-fidelity retention, what can be compressed, and how information across different timescales should work together.
Organizing Memory Like the Human Brain
FreshMem draws inspiration from how the human brain processes temporal information and consolidates memories. Recent experiences are retained with greater fidelity, while older experiences are gradually distilled into semantic summaries. At the same time, continuous experiences are organized into relatively independent events, forming more stable and compact long-term memories.
Based on this principle, FreshMem divides online visual memory into three complementary components:
-
Recent High-Fidelity Memory: A sliding window preserves the latest visual information, enabling the model to capture fine-grained actions, object states, and local changes.
-
Multi-Scale Frequency Memory (MFM): Rather than discarding older frames when they leave the sliding window, FreshMem incrementally encodes them in the frequency domain using the Discrete Fourier Transform (DFT). This preserves a global representation of long-term visual history at relatively low computational cost. High-saliency residual information is also retained to recover important edges and structural details.
-
Spatial Thumbnail Memory (STM): Event boundaries are identified based on semantic similarities between adjacent visual features, allowing continuous video streams to be segmented into event clusters. These clusters are then adaptively compressed according to event duration. When the memory budget is constrained, semantically similar events are merged to create compact, information-dense event thumbnails.
The resulting representations—recent visual details, global historical summaries, and structured event memories—are jointly provided to a multimodal large language model.
Instead of choosing between remembering everything and rapidly forgetting the past, the model can preserve information from different timescales at different levels of fidelity.
FreshMem in One Sentence:
The more recent an experience, the more clearly it is remembered. The further back it occurred, the more concise its representation becomes. Truly important events are preserved as structured memories over time.
Toward Online Understanding for Embodied AI Models
Streaming video is not simply another video-processing task. It can also be understood as a continuous stream of first-person experiences received by an embodied intelligent agent.
People, objects, spatial relationships, and movements captured by cameras continuously enter the model, forming a dynamic context through which the agent interprets the physical world.
The hybrid memory approach represented by FreshMem offers several promising directions for enabling embodied AI models to understand their surroundings continuously:
-
Real-Time Obstacle Avoidance and State Recognition: Recent high-fidelity memory helps the model focus on newly appearing pedestrians, moving objects, and changes in manipulation targets.
-
Long-Horizon Task Continuity: Frequency-based memory preserves an overall representation of how the environment evolves, potentially reducing the loss of previously observed information during extended task execution.
-
Event-Level Experience Organization: Event segmentation and memory consolidation can organize continuous observations—such as entering a room, searching for an object, and completing an operation—into structured, retrievable experiences.
-
Continuous Operation Under Resource Constraints: Memory requirements no longer need to grow directly with video duration, offering a scalable direction for long-term online perception on edge devices and resource-constrained platforms.
It is important to distinguish the demonstrated results from their potential applications. The paper's direct experimental evidence comes from streaming and offline video understanding benchmarks. Applying these mechanisms to closed-loop perception, decision-making, and control in real robotic systems will require further system-level validation.
Embodied intelligence therefore represents an important potential application of this research direction, rather than a capability fully demonstrated by the paper's current experiments.
At present, FreshMem's experimental validation focuses primarily on streaming and offline video understanding tasks. Further system-level evaluation is needed before these memory mechanisms can be integrated into continuous perception, task decision-making, and motion control in real-world robots.
Rethinking Video Learning
This research also offers a broader insight: effective video learning does not necessarily require every frame to be processed with equal importance or at the same level of detail.
Real-world videos contain substantial static redundancy, brief but critical actions, and causal relationships extending over long periods. A more efficient learning process should actively determine how information is preserved, at what temporal resolution, and in what structure.
FreshMem highlights three principles worth further exploration:
-
Non-Uniform Temporal Modeling: Recent information is preserved in greater detail, while older information is progressively abstracted into semantic summaries.
-
Saliency-Driven Retention: Even information from earlier frames can be preserved through residual representations when it contains valuable visual structures.
-
From Frame Sequences to Event Structures: Rather than learning only from continuous sequences of images, models can represent dynamic processes through events and the relationships between them.
This suggests that video data can be more than a long context that a model must process from beginning to end. It can become a continuous stream of experiences that the model progressively absorbs, compresses, and consolidates.
For embodied AI models, this perspective on online learning is particularly important. The physical world does not pause while a model processes information. Memory itself must become an integral part of perception and learning.
Beyond Longer Context Windows
FreshMem is not primarily concerned with extending the context window. Instead, it rethinks how models preserve historical information—moving from passively accumulating more frames toward actively constructing hierarchical, selective, and continuously updated visual memories.
This research connects two increasingly convergent directions.
The first is streaming video understanding, which enables models to interpret visual information continuously under strict causal constraints, understanding the world as events unfold.
The second is embodied intelligence, which requires models to accumulate experiences that can support perception, reasoning, and action in continuously changing physical environments.
FreshMem's results suggest that structured memory could serve as an important technological bridge between these two directions.
As intelligent agents move into the real world, they will need more than powerful visual encoders or larger language models. They will need capabilities that more closely resemble continuous cognition: perceiving the present, understanding the past, and turning every observation into meaningful context for the next action.
Looking ahead, Motion Brain will continue exploring the integration of visual memory with its embodied AI brain technology, further investigating its potential through real-world robotic tasks.
