In multimodal large language models (MLLMs), human memory is not a mere recording but a reconstructive process. Current MLLMs fall short in this regard, as they process images through a frozen visual encoder, produce one-shot text outputs, and discard internal representations. To address this, we present DoYouRemember, a three-stage architecture that introduces reconstructive memory into MLLMs:
- A VQ-VAE compresses images into discrete visual tokens;
- A LoRA-fine-tuned LLM jointly attends to visual and text tokens;
- A Diffusion Decoder reconstructs images from the LLM's hidden states.
On 1,000 3D facial skin texture maps and 99,000 unlabeled facial images, we find that LLM hidden states contain approximately zero recoverable visual information— the same Decoder producing clear reconstructions from VQ-VAE tokens (pre-LLM) generates pure noise from LLM hidden states (post-LLM), demonstrating that the LLM understands images but does not remember them.
Training a shared memory matrix M under backpropagation systematically fails due to gradient cancellation (O(1/sqrt(N)) attenuation). We identify three root causes and show that local EMA updating resolves all three: each image updates only its top-8 slots out of 64, preserving inter-slot diversity. The resulting M (229K parameters, 16x compressed) approaches the VQ upper bound on unseen test images. Scaling to 1,024 slots surpasses it (LPIPS 0.056 vs. 0.071), as M's continuous representation avoids VQ quantization error. We unify these findings under an information-theoretic framework: memory is lossy compression, recall is decompression, and hallucination is an inherent property of lossy decompression rather than a defect.
Blogger's Review: This research introduces the concept of reconstructive memory into multimodal AI systems, challenging traditional understandings of information processing in current models. The DoYouRemember architecture offers new insights for the design of future intelligent systems, warranting further exploration.