Using large language models (LLMs) as judges has become a common, cost‑effective alternative to human evaluation. Because a single judge can be biased, aggregating several judges is the standard remedy. In practice, judge pools contain models with widely varying reliability, while the human‑labeled sets used for validation often contain only a few hundred items or less. Under such conditions, simple per‑item averaging gives weak judges the same influence as strong ones, and confusion‑matrix methods such as Dawid‑Skene must estimate a full error table for each judge from too few items, leading to unstable estimates.
We introduce AGORA (Aggregation Of Reliability‑weighted Assessments), an aggregator built on Item‑Response Theory (IRT). AGORA places every evaluated item on a latent quality scale $\theta$ and characterizes each judge with two parameters:
- a sensitivity $\alpha$ that measures how sharply the judge perceives changes in quality;
- a set of ordered thresholds $\beta_1,\dots,\beta_K$ that define the boundaries between the $K$ rating levels the judge can output.
The model jointly learns judge reliability from all items, without any additional human labels. The likelihood of a rating $r_{ij}=k$ given item $i$ and judge @@@MATH_BLOCK6@@@ is $$ P(r{ij}=k\mid \theta_i,\alphaj,\beta{j}) = \sigma\bigl(\alpha_j(\thetai-\beta{jk})\bigr)-\sigma\bigl(\alpha_j(\thetai-\beta{j,k+1})\bigr), $$ where $\sigma$ is the logistic function.
Experiments on three public benchmarks show that:
- AGORA matches or exceeds Dawid‑Skene when the judge pool is large and heterogeneous;
- Its advantage grows markedly as the validation set shrinks to a few hundred items;
- A label‑free dispersion statistic can predict in advance when trust weighting will be beneficial;
- The learned trust weights correlate strongly with human‑measured judge validity.
Review: AGORA leverages IRT to jointly model judge sensitivity and rating thresholds, enabling reliable weight estimation even with scarce human labels. The approach offers an interpretable and robust solution for automated LLM judge aggregation.