DI-Bench is an automated pipeline that systematically generates in‑domain benchmarks for data‑intelligence (DI) tasks. DI aims to extract insights from massive enterprise data, requiring both analytical computation over structured tables and retrieval of business knowledge. The pipeline first builds an artifact linkage graph linking data tables, dimensions, metrics, and documents, capturing the relationships among these entities. Using this graph, it creates question‑answer pairs that involve structured data together with relevant knowledge; ground‑truth answers are obtained by executing the corresponding queries. An LLM then rewrites the questions and validates consistency. Applying the pipeline to two public datasets yields a benchmark of 731 tasks covering three categories: knowledge retrieval, analytical computation, and rule‑grounded reasoning. To test the benchmark’s discriminative power and difficulty, four models were evaluated. The results show that when a task requires retrieving business rules and incorporating them into the computation, model accuracy drops to only 32%, indicating a substantial gap in current systems.
Review