This paper investigates why vision‑language models (VLMs) can repeatedly self‑correct, say “Wait, I should recheck,” arrive at a wrong answer, yet still report high confidence. We find that verbalized confidence is largely trajectory‑independent in the VLMs and calibration methods we evaluate.
We analyze the phenomenon from three complementary angles:
- Content variation – altering details of the reasoning process and observing confidence changes.
- Token masking – hiding key tokens in the trajectory to test confidence sensitivity.
- Model hesitation markers – leveraging the model’s own “thinking” signals to assess the link between confidence and actual reasoning.
Our experiments show that confidence is insufficiently sensitive to the actual reasoning content, and calibration training can paradoxically worsen this disconnect. Standard calibration metrics such as ECE and AUROC fail to detect the issue. To address this, we propose the Trajectory‑Grounding Score (TGS) in two forms:
- TGS‑self compares confidence with and without access to the model’s own trajectory.
- TGS‑pair tests whether the model assigns higher confidence to correct trajectories than to flawed ones across vision, reasoning, and answer axes.
We introduce TGS‑Bench, a model‑agnostic benchmark suite covering ten tasks, each providing controlled good/bad trajectory pairs. Results reveal a substantial divergence between conventional calibration rankings and trajectory‑grounding rankings, exposing a blind spot in current evaluation practice.
Review: The study systematically uncovers the root cause of confidence‑trajectory decoupling in VLMs and offers a practical evaluation framework, guiding future work on more reliable model calibration.