Code agents continuously evolve within real repositories, and memory systems aim to help them reuse past experience. Because such experience is both diverse and changing over time, it remains unclear how memory systems organize it and how they affect agents. Existing benchmarks lack temporally valid histories or control over the relevance of prior tasks, making reliable answers difficult.
This paper introduces MemDev-Bench, a benchmark for evaluating how memory systems influence code agents across varied repository histories. For each target task it pairs earlier tasks from the same repository that are annotated with relevance, thereby constructing histories with controlled relevance. An automated pipeline makes these earlier tasks executable and verifiable, yielding 4,869 historical tasks covering 555 base tasks.
Systematic evaluation on MemDev-Bench across different development scenarios leads to three main findings. First, memory mainly improves fine‑grained localization and editing, which raises functional correctness. Second, even with related history, memory can cause failures on tasks that would otherwise be solvable; unrelated history further increases such failures, primarily because agents reuse memory without reassessing its applicability. Third, providing agents with raw historical trajectories directly reduces the number of steps compared with using memory systems. Further analysis suggests agents save steps by reusing process information that compressed memories tend to omit.
These results suggest future memory systems should preserve process information and reassess memory applicability against the current task context.
Review