Multimodal benchmarks are pivotal for assessing whether models can perform cross‑modal reasoning. However, recent studies have shown that many benchmarks contain shortcut items that can be solved using a single modality or by exploiting answer‑option regularities, thus failing to reflect genuine cross‑modal inference ability. Existing evaluation approaches either focus on intrinsic item structure or on extrinsic model responses, which are insufficient for a comprehensive diagnosis.
To address this gap, we introduce SRCoD (Structure‑Response Co‑Calibration Diagnosis), a framework for item‑level diagnosis of modality dependence in multimodal benchmarks. SRCoD first estimates a sample‑wise multimodal information structure from the benchmark text and target answers, and uses this signal as a soft structural anchor to condition the item‑response model. The calibrated model yields an item‑level diagnostic profile that quantifies how strongly each item relies on different modality evidence, enabling fine‑grained analysis beyond aggregate accuracy. Moreover, SRCoD provides interpretable attribution of the diagnosis results.
Extensive experiments are conducted under two complementary protocols: benchmark‑refinement oriented validation and human‑labeled item validation. Results demonstrate that SRCoD consistently outperforms state‑of‑the‑art benchmark diagnosis methods.
Review: By co‑calibrating structural cues and model responses, SRCoD offers a fine‑grained, interpretable diagnosis of multimodal benchmarks, furnishing a valuable tool for benchmark design and model evaluation.