Frontier AI models are increasingly deployed in high‑stakes domains such as healthcare, defense, finance, governance, and politics, where expert judgment is essential.
We propose a general methodology that distills human expert judgment into automated evaluation systems that can operate at scale. The approach assembles networks of domain experts, extracts both conclusions and underlying reasoning, and encodes the reasoning into structured editorial standards that automated judges apply continuously.
The methodology builds on a long tradition of expertise research, arguing that independent, expert‑calibrated evaluation is required both epistemically and for public accountability.
We validate the approach on AI news reporting as a first application. For source quality, two independent experts achieve Krippendorff’s $\alpha$ of $0.83$ and $0.69$ on a held‑out test set, while the automated judge attains $F1$ scores of $0.81$ and $0.97$ against expert consensus. For neutrality, experts reach $\alpha = 0.72$, and the automated judge records $F1 = 0.86$.
A controlled comparison shows that expert‑calibrated judges substantially outperform uncalibrated frontier models on every source‑quality metric, demonstrating that encoding expert reasoning is a critical step that prompt engineering alone cannot replace.
We present this benchmark as a proof of concept for a broader evaluation infrastructure applicable to any domain where judgment matters and expertise is scarce.
Review