Abstract
Long-term memory enables LLM agents to reuse past interactions, but raw dialogue histories are verbose and information-sparse. Broad retrieval improves evidence coverage but overwhelms downstream reasoning with noise; compressing at write time reduces noise but irreversibly discards details needed for future queries. We introduce LazyMem, which sidesteps this dilemma by deferring all memory construction to query time.
A lightweight 4B model processes the retrieved candidate pool in overlapping parallel windows, selectively retaining and compressing only query-relevant content. The model is trained through supervised fine-tuning followed by group-based reinforcement learning with a format-gated composite reward that combines a rule-based action signal measuring selection accuracy with an LLM-judged quality signal measuring source faithfulness and query utility.
On the LongMemEval benchmark, LazyMem-4B achieves an LLM-judge accuracy of 0.85 with only 213 memory tokens, 68.7× fewer than retrieval-only, and generalizes to LoCoMo (0.68) without target-domain training, while reducing mean latency over the prior query-time baseline. The 32B variant reaches 0.93, surpassing oracle-context references on aggregation-heavy question types. The code associated with this work is publicly available at GitHub.
Blogger's Review: LazyMem effectively addresses the information overload problem in long-term memory by deferring memory construction, demonstrating how to enhance model selectivity and efficiency in complex reasoning tasks. This innovative approach not only improves model performance but also provides new insights for future memory management strategies.