In video large language models (LLMs), benchmark accuracy is often treated as evidence of visual understanding. We audit this assumption across twenty models with parameters ranging from 200M to 7.8B across ten architecture families. We introduce the Visual Dependency Gap (VDG), which measures the difference in per-question correctness between original-video and black-screen conditions. Paired McNemar tests on MVBench indicate that accuracy and visual dependency are separable: models differ significantly on original video (p = 0.0003) but not on black screens (p = 0.53).
Task-type rankings are stable across models: Attribute Perception is strongly visual, while Temporal Reasoning approaches the language-only baseline. A diagnostic ladder from black screen to single frame, shuffled frames, and original video reveals that frame diversity supplies most of the visual benefit, whereas temporal order contributes almost zero accuracy across sixteen open-weight models. An ablation study varying from 0.5 to 24 FPS rules out sparse sampling as a cause. H.264 experiments further show that stable aggregate accuracy conceals bidirectional question-level answer flips. This diagnostic also generalizes to four API-accessed models, with VDG values ranging from 0.025 to 0.315. These findings motivate the use of VDG as a standard audit for assessing whether video benchmarks measure visually grounded capabilities. Code is available at GitHub.
Blogger's Review: This study highlights the limitations of current video LLM benchmarks, stressing the disconnection between accuracy and visual understanding. The proposed VDG metric offers a fresh perspective for future model evaluations, making it a valuable tool in related fields.