Large language models (LLMs) excel at many narrow tasks, yet querying them individually for millions of related instances can be prohibitively expensive. This work introduces the notion of “bottling,” i.e., converting general capabilities into task‑specific, reusable artifacts that balance answer quality with amortised cost.
The BOTTLED benchmark gives an agent an entire unlabeled workload and requires completion within fixed time, compute, and LLM API budgets. Agents may choose any approach, such as training a small model or writing a reusable program.
Across ten models and three tasks, strong zero‑shot performance does not reliably predict bottling ability. Models with similar zero‑shot scores diverge markedly after bottling; 48 of 60 bottling runs fall below the lower bound of the 95% confidence interval of their model’s zero‑shot performance, and 31 runs underperform two small‑model distillation baselines with the same token budget.
Nevertheless, bottling can yield substantial savings. On a query‑product relevance classification task, Opus 5 retains about 82% of its zero‑shot macro‑F1 while reducing reported cost by roughly 657×. Compared with Jev, a “system‑one” model built for cheap repetitive inference, Opus 5 recovers about 94% of Jev’s macro‑F1 at a quarter of Jev’s projected full‑workload cost.
BOTTLED provides a foundation for evaluating and improving agents’ ability to invest limited resources in reusable solutions for large, repetitive workloads.
Review