Evaluating language models and agents requires jointly measuring benchmark accuracy, judge‑based assessment, runtime performance, tool‑use behavior, and retrieval quality. Heterogeneous interfaces, metric semantics, and trace formats make direct comparison difficult and can hide failures in agentic workflows. We introduce AuditKIT, a source‑available evaluation library that captures all these measurements in a single result representation. It features direction‑aware metric contracts, configuration fingerprints, and extensible metrics and backends.
For agents, AuditKIT provides turn‑aware tool‑call matching, separates parallel from sequential execution, and distinguishes verified task outcomes from trajectory‑level diagnostics. Missing or unobservable evidence is marked as unscored rather than counted as a failure.
In retrieval‑augmented generation, AuditKIT combines ranked retrieval metrics, judged grounding, and deterministic evidence checks. Seven case studies—spanning model comparison, serving runtimes, retrieval, tool calling, and interoperability—demonstrate how answer extraction, runtime behavior, and metric selection can change evaluation outcomes.
Review