In production, LLM agents are re‑evaluated as they evolve, yet running the full benchmark each time is costly. This work examines a analytics agent serving tens of thousands of monthly active users, using 574 historical benchmark runs split chronologically into a calibration phase and a held‑out phase. Four recurring‑evaluation methods are compared: random sampling, historical caching, fixed representative subsets, and IRT‑based adaptive testing. Results show that multidimensional 2‑parameter logistic (2PL) adaptive testing yields the highest score fidelity—executing only 200 questions (38.5% of a full run) adds just 1.03 pp MAE. For operational simplicity, we deployed difficulty‑stratified fixed subsets, demonstrating that they transfer without recalibration to five other agent families and remain stable even when calibration windows shrink to a single day. Drawing from this deployment, we recommend: 1) use difficulty‑stratified fixed subsets for routine monitoring; 2) periodically validate subset representativeness with a small random sample or IRT adaptive test; 3) quickly recalibrate subsets after major model updates to avoid drift.
Review