Modern vision‑language models (VLMs) can answer many image‑grounded questions directly, yet they struggle with complex queries that require fine‑grained visual details or external knowledge. To obtain missing evidence, agentic VLMs invoke tools such as image cropping, image search, and text search. Existing training paradigms mainly evaluate tool use by final answer correctness, leaving evidence acquisition and utilization under‑supervised. This creates two major issues: (i) models often issue redundant or off‑target tool calls that fail to gather necessary evidence; (ii) even when appropriate tools are called, models frequently do not extract the required information from the observations.
We introduce the Necessary Tool‑Evidence Path (NTEP) annotation scheme, which explicitly specifies the essential external evidence and corresponding tool calls for each query. Building on NTEP, we propose NTEP‑R (NTEP Reward), a supervision mechanism that ensures every tool invocation strictly advances the reasoning toward the final solution. Specifically, we reward alignment between the pre‑call intent and a necessary evidence‑seeking goal, and we reward that the summarized information from the post‑call observation matches the necessary evidence. Additionally, a non‑repeated‑goal regularizer penalizes redundant calls that revisit already satisfied NTEP goals.
Extensive evaluations on seven image‑grounded benchmarks show that the 8B‑parameter instantiation, NTEP‑8B, markedly improves both search‑oriented accuracy and tool‑use efficiency within a unified three‑tool framework. These results highlight the critical value of fine‑grained tool‑evidence path supervision for training robust agentic VLMs.
Review