Abstract
Language agents operate in a loop - observe, reason, act - but the memory they reason over is external to this loop: a store queried at most once per turn. We examine the scenario where memory is integrated into the loop, being read and written at every step. The challenge has always been latency: networked stores respond in tens to hundreds of milliseconds, and in-loop retrieval can inflate end-to-end latency by up to 83x when retrieval is costly.
Prior work manages this cost instead of questioning it: serving-layer scheduling hides it, and 'memory-first' designs ration retrieval to once per turn. We argue that latency is a property of the store's location, not the in-loop pattern: an in-process store responds in ~100us, three orders of magnitude below the network regime, collapsing the per-step tax at that speed.
According to the parity principle of the extended-mind thesis, a store fast enough to be constantly and directly available becomes extended working memory, not just a tool the agent consults. The premise is causal: maintaining a fixed per-turn memory latency budget while varying only the store's answer speed, redundant actions increase monotonically with latency - 0.0 out of 12 at in-process speed, 7.2 out of 12 at a 110ms cloud round trip (gpt-5-nano, gpt-5-mini; exact permutation p=0.0079).
We demonstrate this regime end-to-end: across four GPT-5-class models under a bounded window, recall improves from 0/5 to 3.6-4.8/5 with in-loop memory, store ops at p50 80-165us - though an instructed restate-every-reply baseline also solves it perfectly, at a token cost that grows with the working set.
The store never lost a fact in any run (244 of 244 writes kept); every miss traces back to the agent's read policy, not the store. Our measurements also relocate the bottleneck: the dominant per-step cost is embedding (~200-400ms over the network); pairing the in-process store with a small local embedder returns the complete operation to a measured ~40us.
Blogger's Review: This study delves into the critical role of memory in language agents, proposing an innovative approach to significantly reduce latency by integrating memory retrieval into the loop. This not only optimizes memory efficiency but also offers new insights for the design of future language models, especially in real-time task handling. By minimizing network latency, agents can respond more swiftly, enhancing overall performance, which is noteworthy.