Consensus among LLM judges is often taken as strong evidence of correctness, assuming that judges make errors independently. In practice, judges are trained and evaluated in similar ways, leading to correlated mistakes. In our main experiment with ten judges we measured an average pairwise error correlation of $0.21$, meaning the ten judges provide roughly the same statistical information as $3.5$ independent judges. When we evaluated high‑accuracy frontier judges, including models from different providers, the dependency was even stronger. Ignoring shared errors can cause up to $28\%$ of comparisons to falsely claim a significant advantage for one system, while accounting for the dependency removes that claim. The pattern of errors also matters: errors shared by most judges versus errors concentrated in a small subset affect consensus differently and favor different voting schemes. Thus, measuring overall correlation alone is insufficient. We recommend using a small set of trusted examples to estimate judge accuracy and identify shared mistakes, incorporating those shared errors into the analysis, and selecting the voting method based on trusted examples before applying it to new data.
Review