Business intelligence (BI) underpins enterprise decision‑making, and traditional BI workflows typically involve four steps: (1) identifying relevant tables, (2) performing data transformations, (3) building join relationships, and (4) answering business questions. These steps are often complex and time‑consuming, raising the barrier for BI adoption.
Given the strong data‑handling capabilities of large language models (LLMs), we investigate whether LLMs can answer BI questions end‑to‑end without manual preparation. To this end, we harvest a large collection of real‑world BI projects from public sources and manually extract (question, ground‑truth answer) pairs from user dashboards, creating the first benchmark—BI‑Bench—to systematically evaluate LLMs on end‑to‑end BI tasks.
Our evaluation reveals that even state‑of‑the‑art LLMs achieve less than 50% accuracy on BI‑Bench, far below practical requirements.
To address this gap, we propose a tool‑augmented BI‑Agent that decomposes the BI workflow into structured‑data subtasks such as search, join, and transform, orchestrating specialized data‑management methods at each stage.
We also develop a post‑training framework that synthesizes training trajectories from real BI projects, enabling the BI‑Agent to undergo supervised fine‑tuning (SFT) and reinforcement learning (RL) for domain‑specific adaptation.
Experiments show that vanilla LLMs combined with the BI‑Agent gain up to 40 percentage points in accuracy, while the post‑trained BI‑Agent adds another ~30‑point boost, dramatically narrowing the performance gap.
These results highlight the importance of coupling tool‑augmented reasoning with domain‑specific post‑training in complex BI workflows and point to promising directions for future research. Review