As large language model (LLM) agents evolve from answering questions to executing full procedures, failures often hide behind skipped checks, branches, dependencies, or invariants, making the final output appear acceptable. Traditional output‑only evaluation can only judge the result, while trace‑aware judging sees the actions but cannot pinpoint which obligations were active for a given query.
ContractEval is introduced as a diagnostic framework that makes procedural instructions explicit as query‑active obligations and matches them against the model's response or execution trace. This matching turns omitted checks, wrong branches, ordering errors, extra actions, invariant breaches, and output‑contract violations into distinct conformance failure categories.
On a controlled suite of audited procedural contracts, pure output or trace judges miss many injected structural faults; using gold‑standard expected and observed graphs, ContractEval detects and localizes all of them. LLM‑backed extraction retains most of the signal but remains sensitive to model calibration.
Thus ContractEval does not guarantee compliance; it transforms procedural conformance from an implicit property of the final answer into an auditable, explicit check.
Review: The framework offers a practical way to audit LLM reliability in complex workflows, especially where strict business rules must be enforced.