Abstract
While large language models (LLMs) enable strong question answering (QA), budgeted deployment is complicated by nondeterminism and heterogeneous resource profiles (cost, latency, and energy). We present OPTIMIZE, a database-inspired, cost-based optimizer that implements a plan-before-execute paradigm for multi-LLM orchestration.
OPTIMIZE models LLM invocations as physical operators in an execution DAG and, for each question, searches for plans that optimize answer quality (QoA) while trading off financial cost, latency, and energy under user-specified resource constraints. Plans can include sequential operators that pass intermediate answers as context and parallel/blend operators that run models concurrently and merge their outputs.
To search this space without executing each candidate plan, OPTIMIZE uses PERFDB, a statistics catalog populated and refreshed from benchmarks and execution traces, to estimate the QoA and resource costs of both individual operators and composed subplans. Using these estimates, OPTIMIZE performs Pareto-frontier search and selects a final plan based on user preferences. On MMLU-Pro and SimpleQA under user-specified budgets, OPTIMIZE improves average QoA by ~58% and ~41% over baselines at comparable cost, demonstrating that database-style planning yields better quality-resource trade-offs for multi-LLM QA.
Blogger's Review: The OPTIMIZE framework effectively integrates the needs for resource management and quality enhancement in QA processes by modeling the question-answering path as an execution graph. Its innovative statistical estimation method significantly boosts the efficiency of multi-LLM systems and serves as an important reference for the future of QA system development.