Multimodal large language models (MLLMs) can answer knowledge‑intensive visual questions by combining visual evidence from images with facts retrieved from external sources. Yet, they often miss crucial evidence, attending weakly to the textual sentences or visual regions needed for the correct answer. Existing approaches highlight retrieved text or mark visual regions before generation, but they follow a fixed one‑shot policy that cannot adapt to three sources of variation: whether highlighting is needed, how much evidence each example requires, and when different textual evidence becomes relevant as the answer unfolds.\
We introduce Adaptive Relevance‑guided Evidence Allocation (AREA), a training‑free inference‑time method that treats evidence highlighting as an adaptive allocation problem. AREA first emits a single probe token to read visual and textual relevance from fixed backbone layers, then makes three decisions:\
- Whether to intervene – controlled by natural attention coverage and visual sink contamination;\
- How much evidence to expose – determined by relevance entropy;\
- When to refresh text – triggered by peaks in causal context‑attention.\
Across four KB‑VQA and seven standard multimodal benchmarks using nine frozen MLLM checkpoints, AREA achieves the best performance among training‑free highlighting methods.\
Review