Evaluators often output correct labels while reasoning incorrectly, posing a critical risk for agentic systems that rely on evaluations to gate actions, route reviews, or provide training feedback. Conventional evaluation checks only the final label correctness, ignoring whether judgment changes stem from valid evidence, consistent rules, or proper rule applicability. We formalize evaluator reasoning accountability into three core sources: grounds, norms, and authority. Varying these sources yields an eight‑cell counterfactual judgment cube that characterizes different judgment update scenarios.
Judgment receipts are defined as minimal source‑replacement sets that reproduce revised verdicts, thereby explaining the causal path of judgment transitions. We derive certification cost bounds for black‑box evaluators and introduce ReasonBench, a policy and logical reasoning benchmark covering 19,520 cases and 7,200 controls, each equipped with verifiable receipts.
In frozen evaluations, Qwen3‑1.7B achieves 98.41% receipt accuracy while cube prediction scores 96.99%, a consistent 1.42‑point gap validated by Qwen3‑0.6B replication. High standard accuracy masks severe robustness flaws. Meaning‑preserving source permutations reduce valid receipt recovery to 54.8% (direct) and 49.2% (cube) predictions.
Models trained on simple single‑source changes retain 93.75% verdict accuracy but recover only 7.16% of receipts for complex multi‑source updates. Permutation retraining boosts consistency to 96.6% yet worsens cube prediction deficits. Structured counterfactual supervision fails to guarantee robust reasoning. We argue that reason‑aware evaluation must decouple prediction from certification, reporting transformation consistency alongside standard accuracy for trustworthy evaluator auditing.
Blogger's Review: The paper exposes the hidden fragility of evaluator reasoning behind superficially correct outputs and offers a receipt framework that provides actionable explanations for audits. Experiments show that even large models can score high on traditional metrics while falling short on causal consistency, highlighting the need for future evaluation systems to safeguard both outcome and process.