Rubric‑based evaluation is a common practice for assessing the outputs of large language models (LLMs). It decomposes overall quality into several criteria, each receiving a binary judgment (pass/fail) from an LLM verifier. In practice, the certainty of these judgments varies widely; some criteria are judged with high confidence while others remain ambiguous. Existing pipelines simply aggregate all binary judgments with equal weight, implicitly assuming uniform certainty and ignoring this heterogeneity. Consequently, uncertain judgments are fed directly into reward estimation, potentially providing misleading signals for downstream reinforcement learning (RL).
To address this limitation, we introduce Uncertainty‑aware Rubric Weighting (URW), a training‑free framework that quantifies criterion‑level uncertainty via conformal calibration. The procedure consists of:
- Building a fixed pool of judgment templates for each criterion and generating paraphrased variants.
- Mapping the LLM’s binary judgment to similarity scores against the template pool, yielding a soft uncertainty signal.
- Converting this signal into a per‑criterion weight $w_i$ using split‑conformal $p$‑values, which satisfy the coverage guarantee $$\Pr\{\text{true error rate} \le w_i\} \ge 1-\alpha$$ for a chosen significance level $\alpha$.
Weights $w_i$ decrease as uncertainty grows, automatically reducing the influence of unreliable criteria in the final score. URW requires no additional training; only template construction and a one‑time conformal calibration are needed.
Extensive experiments on several public LLM benchmarks show that:
- URW improves pairwise preference accuracy by roughly 2%–4%.
- Incorporating URW into reward models yields about a 1.5% gain in downstream RL alignment scores.
These results demonstrate that explicitly modeling judgment uncertainty and weighting accordingly can substantially enhance the reliability of rubric‑based evaluation and the quality of downstream alignment.
Review