NeFut Logo NeFut
Admin Login

[CS.AI] C$^3$PO: Evaluating Cross-Modal Composition and Counterfactual Performance

Published at: 2026-08-07 22:00 Last updated: 2026-08-08 01:08
#AI #Machine Learning #Multimodal

Current Multimodal Large Language Models (MLLMs) can process diverse sensory inputs, yet their reasoning remains heavily biased toward a dominant modality, resulting in brittle cross-modal reasoning. We introduce C$^3$PO, a benchmark of 3,404 samples spanning video, audio, image, and text, evaluating two abilities: information composition (fusing dispersed evidence) and counterfactual conflict (resolving deliberate contradictions). C$^3$PO's paired IC/CC structure and four-tier design enable targeted diagnosis of when and why cross-modal reasoning fails. Built through a fully automatic pipeline using 25 logically grounded templates, C$^3$PO reveals that while humans achieve 88.64% accuracy, the best model (Gemini-3.1-Pro) reaches only 73.17%, with open-source models collapsing under conflict. Through attention probes, we find 86-95% of failures stem from modality dominance: models commit to one modality while ignoring contradictory evidence, concentrating 87-95% of attention on text. Mid-layer attention entropy predicts correctness-sustained exploration succeeds, premature collapse fails. The 56-point accuracy gap between equally complex templates reveals that performance depends on modalities' structural roles in conflict resolution, not combinations. These findings show multimodal perception does not guarantee robust reasoning; architectures must enable sustained cross-modal attention to avoid premature collapse. Blogger's Review: The C$^3$PO benchmark reveals the limitations of current multimodal language models, highlighting the importance of cross-modal attention and conflict resolution capabilities. Future research should focus on developing more robust cross-modal reasoning models and architectures to tackle complex real-world problems.

Original Source: https://arxiv.org/abs/2608.05381

[h] Back to Home