Researchers increasingly employ large language models as judges (LLM‑as‑a‑judge) to evaluate model outputs, yet no standard design guidelines exist. We systematically compare ten reasoning models on two tasks: scalar rating of sentence sentiment and toxicity on a 1‑7 scale (over 500 items per category) and binary accuracy classification of question‑answer pairs (n=600).
For the rating tasks, judges disagree significantly with human ground truth, but the mean absolute deviation is only 0.11 points, indicating most judges are reliable; toxicity judges even outperform conventional classifiers. The binary task achieves an average accuracy of 96.5%.
Design choices can cause notable shifts: changing only the rating scale can alter measured bias by up to 0.93 points; in the classification task, detailed prompts reduce leniency by 28.9%, and switching models can cut leniency by as much as 56.1%. Counterintuitively, reducing reasoning effort does not affect accuracy or leniency. Across both tasks, model identity accounts for the largest variance.
These findings suggest that while LLM judges are broadly trustworthy in aggregate, prompt formulation, rating scales, and model selection remain meaningful sources of variance. This study serves as a methodological reference for building more robust and reproducible LLM‑as‑a‑judge pipelines.
Review