Multimodal long‑document QA requires models to retrieve sparse evidence across pages and modalities, understand heterogeneous content, and reason over the collected clues. Even state‑of‑the‑art vision‑language models often miss such evidence and cannot perform cross‑modal reasoning in a single pass. Existing document agents add multi‑step interaction but still rely on predefined workflows or coarse‑grained actions, limiting flexibility and granularity.
DocPilot is an agentic document framework that orchestrates evidence seeking via page‑level navigation and fine‑grained evidence localization, and integrates local visual inspection, code‑based numerical reasoning, and cross‑modal reasoning for deep multimodal understanding. The core idea is to expose these capabilities as callable sub‑tools, allowing a small language model (SLM) to schedule them through natural‑language instructions and perform flexible multi‑step reasoning.
To transfer these abilities to small models we propose curriculum‑based agentic training. The curriculum starts with single‑ability learning and gradually introduces adaptive orchestration, supervised by data‑grounded reasoning traces and executable interaction trajectories. This enables compact models to acquire complex document‑solving strategies.
Across MMLongBench‑Doc, DocBench, FinRAGBench‑V, and MADQA, DocPilot yields strong gains with frontier VLM backbones, while DocPilot‑4B achieves competitive performance despite a far smaller parameter count, demonstrating effectiveness and generalization across diverse long‑document reasoning settings. Code is released at https://github.com/HKUSTDial/DocPilot
Review