Abstract
Recent unified multimodal models indicate that a single architecture can jointly perform vision/language understanding and image generation/editing. However, they repeatedly feed all historical visual and textual inputs into a shared context window, limiting long-horizon multimodal dialogue due to visual token explosion and unreliable cross-turn referencing.
We propose a Cognitive-structured Multimodal Agent that externalizes visual information into an Episodic Visual Memory and selectively reactivates relevant episodes during reasoning. The agent consists of:
- Perceptual Abstraction Engine for structured visual abstraction;
- Cognitive Retrieval Engine for cross-turn memory retrieval;
- Multimodal Executive Controller for autonomous task inference and action planning.
To address the lack of turn-level retrieval supervision in existing datasets, we developed a Unified Scenario Engine that programmatically generates structured multi-turn conversations with fine-grained retrieval annotations, enabling reinforcement learning to optimize abstraction and retrieval policies. We also constructed a long-horizon visual-dialogue benchmark stratified by difficulty to evaluate episodic visual recall.
Our 8B agent achieves 91.4% retrieval accuracy over 20-turn sessions, surpassing 32B baselines by +8.2% while nearly halving per-turn inference time (from 23.1s to 12.7s). Furthermore, we present the Cognitive-structured Multimodal Agent Harness (CMA-Harness), a tool-augmented deployment of the same cognitive structure integrating persistent multimodal memory, web access, image generation/editing/composition tools, and OpenAI-compatible serving.
Structured memory and modular decision-making offer a more scalable, efficient paradigm for long-horizon multimodal agents than monolithic parameter scaling.
Code
Project Page
Blogger's Review: This research opens new avenues in the multimodal dialogue system domain by introducing episodic visual memory and modular design, significantly enhancing efficiency and accuracy in long-horizon dialogues. Its potential in real-world applications deserves attention, and future studies could further explore optimizing the synergy between memory retrieval and task inference.