A persistent agent must decide what information to retain as new data arrives and what to surface when a query is issued. Existing memory evaluations often conflate these two decisions by comparing methods that differ in both retention and selection policies.
We introduce a streaming‑recall benchmark that spans retention rules and selection rules, evaluating every condition on the same 300 seeded episodes to ensure a fair comparison.
Results show that, with access fixed, query‑aware selection improves required‑fact recall by 15.5 percentage points (95% CI: 12.8–18.2). In contrast, a mixed comparison that also changes history access reports a 68.7‑point advantage, of which 53.2 points are attributable to the access change.
Under bounded retention, query‑aware, dense, and oracle selection all reach the retention ceiling. All 319 observed failures in the bounded‑recency condition are caused by eviction rather than ranking errors. As targets move sufficiently far into the past, recall drops to 0%.
Repeating the evaluation on SQuAD preserves the retention ceiling and demonstrates that dense retrieval can outperform lexical retrieval on natural text.
These findings suggest that bounded‑memory evaluations should keep access fixed and report retention and selection performance separately.
Review