In today's business landscape, LLM-based agents are increasingly deployed to address optimization problems. However, existing benchmarks primarily assess these agents on pre-structured mathematical formulations, overlooking the critical challenge of translating complex business requirements into accurate models and solving them efficiently. To address this, we introduce Opti-Agent-Bench, an end-to-end benchmark that evaluates LLMs across the entire optimization R&D pipeline, from understanding business-language descriptions to mathematical modeling, algorithm selection, code implementation, and solution report generation.
Our design rests on three pillars:
- Business-Semantic Authenticity: Anti-template traps that defeat pattern matching.
- Modular Evaluation: Cross-module consistency checking across Problem Understanding, Formal Modeling, Implementation, and Reporting.
- ORAC Bi-level Validity Framework: Ensuring task quality and scoring integrity simultaneously.
Across several industrial-scale tasks spanning integer programming, robust optimization, stochastic programming, and non-convex optimization, we expose critical failure modes of current models, including constraint omission, model-code inconsistency, and report-implementation divergence, which remain invisible under conventional single-metric evaluation.
Blogger's Review: The introduction of Opti-Agent-Bench offers a fresh perspective on the application of LLMs in complex business optimization problems, emphasizing the importance of the entire process from business requirements to solution generation. This real-world scenario-based evaluation method is sure to propel further advancements in the optimization field.