In automated evaluation, LLM-as-a-judge has become the standard, yet its high costs, significant latency, and opaque decisions undermine its scalability and reliability. To address these issues, we propose a simple and efficient alternative: program distillation.
Instead of prompting an LLM at evaluation time, we distill its decision logic into a committee of programs that can score candidates directly. These programmatic judges offer transparency, are easily inspected or edited, and eliminate per-sample API costs. Building on this notion, we introduce the PAJAMA system, which synthesizes programs as judges, aggregates their decisions into a joint verdict, and incorporates a fallback mechanism to selectively escalate low-confidence cases to an LLM.
Across five datasets and four model families, we demonstrate that programmatic judges can match the performance of a 13B-size LLM judge. When using program outputs as routing signals, PAJAMA improves both accuracy and throughput and advances the Pareto frontier. Beyond evaluation, programmatic judges produce cheap and effective reward signals: on RewardBench, a reward model distilled from programs' verdicts outperforms one trained on a proprietary LLM's labels at two orders of magnitude lower API cost.
Blogger's Review: This paper presents a novel approach to automated evaluation through program distillation, showcasing significant advantages in cost reduction and transparency. The design of the PAJAMA system highlights the potential of programmatic judges in practical applications, warranting further attention and exploration.