NeFut Logo NeFut
Admin Login

[CS.AI] Blind to the Pivotal Vote: Aggregate Independence Metrics Miss Where Verification Actually Helps

Published at: 2026-08-10 22:00 Last updated: 2026-08-11 02:05
#Machine Learning #LLM #Artificial Intelligence

A recent study has shown that LLM judge panels are a standard evaluation tool, but prior work reports highly correlated panel errors: nine judges provide roughly the effective information of two independent ones, and aggregation closes only a small fraction of the gap. A natural remedy--a signal from a different evidence source, e.g., executing a test suite--produced no distinguishable change in the panel's effective-vote count at scale (-0.04, 95% CI [-0.10, +0.02]). Aggregate dependence and conditional decision utility are different questions. Elementary majority arithmetic fixes the affected set for single-ballot substitution: only decisions with a one-vote margin can change. The empirical question is whether panel error rates rise and useful substitutions concentrate there. They do: the entire accuracy gain concentrates on these pivotal queries, where it is large (+10.4 to +23.3 percentage points across three headline configurations), and is exactly zero elsewhere. We confirm the pattern across three code benchmarks and four panel sizes (a 9-judge extension and 56 dependent subsampling checks, gain +6.5 to +16.1 percentage points). On HumanEval+/MBPP+, a majority-side replacement rule raises overall accuracy from 82.44% to 85.62% while invoking the signal on 16.2% of queries; signal-only remains stronger at 87.60%. Thus population-level dependence diagnostics and margin-stratified utility are complementary, and the affected-set characterization yields a call-reduction rule for any specified single-ballot substitution policy. Blogger's Review: This article reveals that aggregate independence metrics may miss where verification actually helps, especially in pivotal votes. The study's findings highlight the importance of considering margin-stratified utility when evaluating LLM judge panels, as accuracy gains are concentrated in these critical queries.

Original Source: https://arxiv.org/abs/2608.06940

[h] Back to Home