Abstract
Effective memory is crucial for large language model (LLM) agents, yet constructing it effectively remains a challenge. A memory-construction policy decides what information to extract, store, update, compress, or discard as interactions accumulate. Heuristic memory methods rely on subjective, task-specific rules, which can misalign with downstream objectives and limit cross-task adaptability.
In contrast, RL-based methods learn from task feedback but mainly use outcome- or module-level rewards. These coarse signals indicate task success but cannot identify which intermediate memory contents support the final answer, creating a fine-grained credit-assignment bottleneck.
However, constructing such process feedback is prohibitively difficult because intermediate memory decisions lack unique ground-truth targets, while the appropriate credit varies with the agent's uncertain reasoning trajectory and therefore cannot be specified in advance. We propose AttriMem, an attribution-guided process-feedback framework for learning memory-construction policies. AttriMem augments the global outcome reward with local rewards derived from token-level contributions to the final answer.
Experiments on long-horizon dialogue question answering show that AttriMem outperforms retrieval-based, heuristic, and RL-based baselines, generalizes across benchmarks and answer models, and stabilizes RL optimization.
Blogger's Review: AttriMem addresses the credit assignment issue in memory construction by introducing a local reward mechanism, providing a fresh perspective on efficient memory management for large language models. Its successful implementation could propel further research in multi-task learning and intelligent agent domains, showcasing broad adaptability in complex tasks.