Evaluating task‑oriented dialogue agents requires more than checking linguistic fluency; each turn must be verified to advance the underlying workflow state correctly. Conventional holistic LLM judges treat the whole context as a single unit and need one or more full‑model calls per turn, often missing state‑level errors.
SAGE (State‑Grounded Abstention‑Aware Evaluation) compiles a workflow specification together with per‑turn state diffs into atomic, schema‑grounded criteria. Each criterion is routed through a cascade of symbolic rules and encoder/NLI verifiers that abstain rather than guess when uncertain, and the individual verdicts are aggregated into a turn‑level decision with an evidence trace.
The recommended operating point, SAGE‑Core, decides 81%–91% of criteria using only the compiler, symbolic rules, and on‑device encoders, incurring essentially zero paid LLM cost. SAGE‑LLM adds an optional focused‑LLM fallback for open‑class criteria.
Across four slices covering MultiWOZ, Schema‑Guided Dialogue, and ABCD, no evaluated LLM‑as‑judge baseline—including a state‑aware GPT‑4.1 judge and cheaper GPT‑4.1‑mini variants—significantly outperforms SAGE‑Core on any slice. The GPT‑4.1 G‑Eval judge costs about $4.7$–$8.0 per 1,000 turns, whereas SAGE‑Core’s cost is effectively $0.
A two‑annotator human audit (n=200, $\kappa$=0.94) confirms strong label fidelity on transcript‑visible failure classes; excluding the weak‑salience IUV class, SAGE‑Core is statistically tied with the strongest LLM judge, and it honestly scopes ignored‑user‑value as a state‑consistency signal with limited broad‑human salience. We also analyze construct‑validity limits arising from injected failures and partial symbolic circularity.
Review