Abstract
Human episodic memory supports retrieval of experiences that unfold over extended timescales, yet the computational mechanisms underlying this ability remain debated due to limited mechanistic accessibility in long-term memory experiments. Long-context language models (LLMs) may offer promising ways to reveal plausible computational mechanisms driving such retrieval.
In this study, we investigate whether and how LLMs capture core behavioral signatures of episodic memory via a temporal order memory task. Using a new dataset of human behavior based on memory of a full-length novel, we find that models exhibit the same characteristic distance effect observed in humans on this task.
Next, we apply long-context mechanistic interpretability analyses to uncover how models solve this task and find that model performance relies on a one-dimensional temporal code reinstated during retrieval by a single time-reinstatement attention head. These findings support temporal context reinstatement as an important mechanism for episodic-like temporal-order memory in LLMs, offering new insights into how temporal aspects of long-term episodic memory may be instantiated in both artificial and biological systems.
Blogger's Review: This paper uncovers the computational mechanisms behind episodic memory in long-context language models, highlighting the significance of temporal context reinstatement. This insight not only provides new directions for model design but also sheds light on understanding human memory processes, holding substantial theoretical and practical implications.