Abstract
An LLM-as-judge score can change even when the candidate responses remain fixed, simply due to a change in the evaluator. We treat this evaluator-replacement ambiguity as a measurement-validity problem. Across four judgment datasets, we compare two upgrade paths available in practice: scaling Qwen3 dense judges from 1.7B to 32B parameters and transitioning across MiniMax M2-M2.7 released APIs.
The main finding is that judge upgrades are not interchangeable: only the upgrade from Qwen3 1.7B to 4B yields a robust adjacent gain, while adjacent releases of MiniMax do not. Stronger judges reduce but do not eliminate position and verbosity bias. Repeated-sample juries add little value when errors are correlated. Structured debate can substantially shift decisions, but without parsers and fallback logs, those shifts cannot be attributed to deliberation.
We argue that LLM-as-judge reports should include dataset slices, bias probes, error-dependence estimates, and protocol audit trails.
Blogger's Review: This article examines the reliability of LLMs as judges, highlighting how changes in evaluators can impact outcomes. By comparing the performance of different models, it provides crucial insights into model selection and the transparency of the evaluation process, suggesting that future reports should be more detailed to ensure the credibility and effectiveness of results.