When an agent is deployed, teams need to know if it truly fulfills its role, not just whether it throws errors. Conventional evaluation is usually a one‑off benchmark run, trace inspection, or a single score, lacking self‑validation and feedback to the agent. We introduce closed‑loop evaluation, modeling evaluation as performance estimation under biased raters and censored outcomes, and requiring every conclusion to trace back to atomic data and be computed by versioned rules. The approach spans four analysis levels—span, trace, conversation, and engagement—measures along a spine of Execution, Substance, Trajectory, and Results, treats conduct as a continuous constraint, and employs Provenance and Adjudication components to certify evidence and evaluators. Evaluator agents are generated by an Agent Factory, each equipped with a domain‑knowledge pack and an adapter harness. Two loops close the evaluation: an inner loop audits the evidence, rules, and metrics themselves; an outer loop uses per‑component regression checks to drive agent improvement. We provide an automated pipeline and demonstrate the method with the business‑analysis agent SharpInsight, also porting it to public conversation data. Results show that problems differ across levels and that evidence‑and‑rule judgment is more reliable than holistic scoring (ICC 0.86 vs. 0.81; kappa 0.81 vs. 0.63).
Review