Traditional visual grounding is treated as a one‑shot mapping from a complete referring expression to a visual target. In real‑world scenarios, however, target information is often incomplete or ambiguous and must be refined through interaction. To address this gap, we introduce a controlled evaluation framework that benchmarks large vision‑language models (LVLMs) on interactive visual grounding. The framework varies (1) how much target information is given upfront and (2) how much must be acquired via dialogue. Experiments span four human‑grounded visual contexts and four interaction protocols.
Across all settings, current LVLMs fall significantly short of task‑level human baselines. Interaction helps when follow‑up questions can refine or repair an initial description, but performance drops sharply when no initial description is provided and the model must acquire the target solely through questions, indicating that proactive, question‑driven grounding remains difficult. Moreover, LVLMs are poorly calibrated, often reporting confidence scores that exceed their empirical accuracy.
Follow‑up studies confirm these patterns across different description sources (human vs. AI), reasoning depths, repeated interactions, description providers, and visual contexts. In sum, interactive visual grounding remains a challenging problem that demands robust visual matching, information‑seeking, and synthesis capabilities.
Blogger's Review: This work systematically exposes the shortcomings of LVLMs in realistic, interactive settings and points to the need for better information‑acquisition and self‑calibration mechanisms in future research.