LLM judges are widely employed to assess agentic systems that can call external tools, yet their reliability on structured, dependency‑driven workflows has not been systematically examined. To fill this gap we introduce AgentJudgeBench, the first benchmark that systematically studies LLM‑as‑a‑judge reliability over workflow directed acyclic graphs (DAGs). The benchmark contains 3,808 instances covering six DAG topologies and three difficulty tiers, evaluated with five generators (3B‑70B open‑weight models and GPT‑5.4) and six judges (20B to frontier scale) under paired with‑ and without‑ground‑truth conditions.
Results show that judge alignment degrades monotonically with task difficulty; the degradation is about 1.5× faster when ground truth is absent. For hard queries without ground truth, all six judges converge to a narrow 77%‑82% alignment band, indicating a structural ceiling driven primarily by difficulty that model capacity alone cannot overcome. Exposing ground truth is not uniformly beneficial: it lowers alignment for GPT‑5.4 by 1.5 pp and for Gemini‑2.5‑Pro by 3.9 pp, reflecting over‑anchoring. Among mitigation strategies, chain‑of‑thought reasoning and higher temperature have negligible impact, whereas structured evaluation rubrics can improve alignment by up to 6.5 pp, though the gains do not generalize uniformly across judge‑generator pairs.
With ground truth, QwQ‑32B best matches the programmatic reference; a human validation study identifies GPT‑OSS‑120B as the most human‑aligned judge; without ground truth, frontier judges only marginally outperform others and remain constrained by the same ceiling. Overall, the findings expose fundamental limits of current LLM judges and provide practical guidelines for reliable evaluation in agentic systems.
Blogger's Review: This work highlights that simply scaling up models or tweaking temperature cannot fix judge reliability; the key lies in pairing structured rubrics with the intrinsic difficulty of the task.