AI agents are increasingly deployed to automate knowledge work over large collections of unstructured data. Such automation demands repeatability: when the underlying evidence does not change, the categories, priorities, and counts produced by the same task should remain materially stable, even if each individual answer appears plausible. We introduce a repeat‑run evaluation framework that first aligns semantically equivalent categories and then focuses on two operating metrics: theme churn, the normalized change in the returned category set; and volume disagreement, the change in counts for categories that persist. Experiments cover three recurring customer‑feedback tasks across eight frontier models, with corpus sizes ranging from 100 to 5,000 records, multiple prompts, and three execution designs: raw generation, taxonomy‑free hierarchical decomposition, and a taxonomy‑grounded agent (TGA) that uses persistent themes, sub‑themes, and record‑level predictions. With Claude Opus 4.8 on a fixed 1,000‑record corpus, TGA reduces theme churn by 86%–88% and yields zero volume disagreement for matched themes. The taxonomy‑grounded agent outperforms every raw model at all corpus sizes and retains its advantage when theme‑matching thresholds are tightened or loosened. Although evaluated on customer feedback, the framework is applicable to repeated synthesis of other unstructured corpora such as financial reports, legal documents, incident records, and scientific literature. Overall, grounding outputs in a taxonomy markedly improves consistency and repeatability for recurring knowledge work.
Review