Aggregating heterogeneous vision‑language models (VLMs) can boost multimodal reasoning, yet neither an individual model’s confidence nor the confidence of the aggregated answer reflects system‑level reliability. We introduce CUSP (Collective Uncertainty through Semantic Opinion Pooling), a training‑free uncertainty quantification framework. CUSP maps the responses of multiple VLMs into a shared semantic response space, pools them into a pooled semantic opinion, and emits two complementary system‑level signals: collective uncertainty—the dispersion of the pooled opinion, and Jensen‑Shannon divergence (JSD)—the conflict among model‑level opinions. Within this pooled opinion, the unnormalized collective entropy decomposes exactly into the mean of the models’ individual semantic entropies plus the JSD, i.e. $$H_{collective}=\frac{1}{M}\sum_{i=1}^{M} H_i + D_{JS}$$, separating total dispersion from model conflict. CUSP requires neither token logits nor calibration labels, so it works with both open‑weight and commercial VLMs. In static multi‑VLM ensembles, collective uncertainty is the strongest signal in the small‑model regime (AUROC 0.764, AUARC 0.889), outperforming majority voting and naive selection by 4.7‑15.8 points and widening its margin as the ensemble grows; in the commercial regime, JSD is strongest (AUROC 0.819, AUARC 0.910) and ranks hard‑answer model conflict with AUROC up to 0.982. The pooled prediction improves accuracy over the average single model by 5.6‑13.0 points. Over the full trajectory of a multi‑step, multi‑agent system, sub‑agent collective uncertainty ranks system failures above chance (AUROC 0.619) and provides the best abstention ordering among evaluated signals (AUARC 0.699).
Review