The usefulness of long‑term memory in large language model (LLM) agents has so far been measured by conversational recall benchmarks such as LoCoMo and LongMemEval, which only test question answering over dialogue history and ignore whether remembered facts actually change a tool‑using agent's behavior. To fill this gap we introduce MERIT (Memory Evaluation for Realistic Instrumented Tasks), a benchmark and harness that quantifies the marginal utility of memory for task‑executing agents while explicitly accounting for cost. MERIT offers episodic tool‑use tasks in three domains; each task’s dependence on earlier‑episode facts is verified by an automated leak check. Difficulty increases along a ladder ending with updated‑fact recall, and we include controlled memory corruption experiments. Every memory read/write is metered for both token usage and dollar cost. Experiments span 23,440 scored episodes (total $42.57), featuring a two‑generation pilot on gpt‑4.1‑mini and a preregistered 3‑model × 3‑seed grid (GPT‑4.1, Claude Haiku 4.5, with memory side fixed). Memory lifts dependent‑task success from a leak‑verified floor of 0.00 to 0.55‑1.00. Retrieval of updated facts is unstable: embedding‑based search varies between 0.30‑0.95 across models (max seed gap 0.45), while write‑on‑retrieve stores—both a structured fact store and LLM summarization—remain at 0.70‑1.00; a hybrid approach performs worse than the fact store alone. A latest‑generation spot‑check with Claude Sonnet 5 under a clean full‑replay control reproduces the pattern. Swapping memory implementations can shift task success by up to 60 points, and full replay is never cost‑effective: the best condition per domain yields only 2.7‑3.9× its marginal utility per dollar. We release the benchmark, harness, and all traces.
Review: MERIT’s fine‑grained cost accounting reveals the real impact of memory on tool‑using agents, offering a more practical evaluation than traditional dialogue‑recall metrics.