Retrieval‑augmented generation (RAG) grounds large language models (LLMs) in external corpora. Conventional agentic RAG can perform iterative search, but it only feeds the model isolated text chunks, lacking document hierarchy, which makes it hard to tell truly relevant evidence from chunks that merely resemble the query. Structure‑aware approaches such as PageIndex exploit a document’s table of contents or sitemap to guide retrieval, yet they require the whole structure to fit into the LLM context and cannot scale to massive corpora; moreover, they commit to a single document during retrieval, so a wrong choice cannot be recovered.
RIT‑RAG (Retrieval‑Induced Tree RAG) builds a tree for each document offline from its TOC or sitemap. At query time, a broad set of chunks is retrieved first; their positions in the original trees are then used to induce manageable sub‑trees, possibly spanning multiple documents. An LLM agent navigates these sub‑trees, selectively reading promising nodes and reformulating the query when needed. In short, retrieval tells the system where to look, while the agent decides what to read.
We evaluate on financial, scientific, and customer‑support benchmarks, comparing against vanilla RAG, graph‑based RAG, and other agentic baselines. RIT‑RAG achieves the highest answer accuracy across all benchmarks. On our newly introduced EntQABench—2.84 million technical‑documentation webpages—RIT‑RAG improves accuracy by 6.8–11.4 points over the strongest baseline, and the gains hold across three different LLM sizes.
Core technical components
- Offline tree construction: parse hierarchical TOC/sitemap into nodes representing sections and subsections.
- Retrieval‑induced sub‑trees: use retrieved chunk positions to cut out sub‑trees, preserving contextual continuity while keeping the search space tractable.
- LLM agent navigation: perform depth‑first or heuristic search on the sub‑tree, dynamically decide which nodes to read, and rewrite the query when necessary.
Review: RIT‑RAG’s ability to map retrieved chunks back to document hierarchy gives it a dual advantage—structure‑aware retrieval plus intelligent reading—resulting in a substantial boost in QA performance on large‑scale technical corpora.