LLM judges are increasingly used as multi‑member panels, and disagreement among members is often taken as a sign of an unclear rubric. We introduce a label‑free procedure that converts such disagreement into rubric repairs.
The procedure first decomposes a rubric into sub‑criteria, computes cross‑judge disagreement for each sub‑criterion on a development split, and ranks them. Only the most contested items are rewritten by an editor model. A subsequent stage moves text‑checkable sub‑criteria to deterministic checkers; in our evaluation this stage transferred none. Human labels are never used for selecting or editing clauses, they are used solely for evaluation.
We pre‑registered four hypotheses and evaluated six rubric conditions with a panel of five judges from three providers on SummEval, the HealthBench meta‑evaluation, and USR Topical‑Chat, totaling 129,770 judge requests (excluding retries).
On SummEval the procedure raised inter‑judge Krippendorff's $\alpha$ by $0.069$ (95% CI $0.046$ to $0.092$) over the original rubric and by $0.074$ over a control that edited an equal number of randomly chosen clauses. Agreement with expert ratings showed no detectable change (Kendall $\tau$ difference $-0.007$, 95% CI $-0.037$ to $0.021$). On HealthBench and USR neither inter‑judge agreement nor human agreement met the preregistered effect criteria.
In a preregistered secondary comparison, human agreement after editing the contested clauses was lower than after editing control clauses in point estimate, though no interval excluded zero. Selecting clauses by disagreement improves consensus among judges but does not bring them closer to human annotators.
We will release code, frozen rubrics, item identifiers, and judge scores.
Review