Automated assessment of clinical AI outputs is essential, yet large language model (LLM) judges are expensive and slow at scale. We compared Jev, a decision language model that produces structured judgments, with frontier and open‑weight LLM judges across four use cases.
On 118 pediatric answers, Jev's agreement with physicians' ordinal quality ratings (Gwet's AC2 0.50, expected 0.53) matched or exceeded seven of twelve LLM judges, including frontier models Claude Opus 5.5 (0.49) and GPT‑6 (0.38).
For binary HealthBench criteria graded by physicians, Jev performed on par with four of six LLM judges.
In MedHELM's open‑ended tasks, Jev's concordance with a three‑LLM jury approached or surpassed the jury's internal agreement.
On the HealthBench Professional benchmark, Jev matched the reference judge GPT‑5.4 on 91% of criteria, comparable to other judges (91‑93%).
Cost analysis showed frontier judges cost 73‑294 times more than Jev and required 7‑57 times longer processing.
Overall, Jev may enable efficient, low‑cost clinical AI evaluation at scale.
Review