Answering complex questions over long documents often requires integrating evidence that is naturally dispersed across distant passages. For instance, in an incident report, the operating condition, design flaw, and missed safety check may appear dozens of sections apart; in a novel, a character's true motive may only surface through scenes far removed from the moment it becomes relevant.
This source-internal evidence integration is central to real-world long-document analysis, yet existing benchmarks largely sidestep it. Current methods like needle probes, planted facts, and reverse-engineered multi-hop chains embed evidence that may differ from the host text in distribution, placement, or register, making it unclear whether strong performance reflects genuine source reasoning or distributional artifacts.
We introduce WILDTRACE, a benchmark of 481 tasks over 214 naturally occurring long-form sources such as technical incident reports and lesser-known literary narratives, where all evidence trails arise from the document's own causal, temporal, and narrative logic. Drawing on Pearl's causal hierarchy and prior multi-hop reasoning typologies, we define seven source-internal evidence geometries that characterize the distinct relational demands of analytical reading in long documents.
A source-first construction pipeline mines candidate trails from document structure before writing questions; each item then undergoes multi-stage validation covering clue necessity, answer groundedness, rubric fidelity, contamination resistance, and answerability. As models are increasingly entrusted with real-world high-stakes analytical tasks, this gap between accessing information and reasoning over naturally dispersed evidence emerges as a defining challenge for the next stage of long-context research.
Blogger's Review: The introduction of the WILDTRACE benchmark addresses a significant need in long-document reasoning, emphasizing the necessity of integrating source-internal evidence. Through systematic task design and validation processes, it lays a solid foundation for future research, advancing the understanding of long-document analysis. This work is not only significant for academia but also provides robust support for practical applications.