Abstract
Large language models and LLM-based agents are widely used as personal chat assistants, enterprise copilots, and autonomous workflow agents. Memory (the ability to retain, access, and reason over information accumulated over long contexts and multiple interactions) plays a crucial role in determining the reliability of any agent. We introduce RECON (Reasoning over Extended Contexts with Obfuscated Narratives), a benchmark for evaluating compositional reasoning over long contexts.
RECON spans 24 case files across three domains (criminal, medical, and financial), each ranging from 50k to 100k tokens, and tests agents on six memory intensive tasks: reconstructing multi-hop evidence chains, propagating cascading invalidations, resolving source conflicts, counterfactual reasoning, satisfying temporal constraints, and temporal fact retrieval.
Recent memory benchmarks evaluate whether agents can retrieve scattered facts or detect if a fact has changed, whereas RECON evaluates what happens after the change, whether agents can trace which downstream conclusions are affected, which survive through independent support, and how alternative timelines would have unfolded.
Our evaluation reveals substantial limitations across current architectures: even the strongest non-Oracle system reaches only 22.4% accuracy, with retrieval and reasoning each surfacing as challenges.
Blogger's Review: The introduction of the RECON benchmark provides a powerful tool for assessing agents' reasoning capabilities in complex scenarios, especially in handling long contexts. The current performance of agents shows significant shortcomings, indicating a need for increased research efforts in designing more sophisticated memory mechanisms.