Large language models (LLMs) are increasingly employed as assistants for statistical and data‑science tasks, yet most existing evaluations assume that the analysis target is already defined. In real scenarios users often present informal goals and heterogeneous data, leaving the model to infer the implied statistical task and identify relevant variables. We formalize this upstream step as Statistical Problem Formulation and decompose it into two subtasks: Statistical Problem Classification and Variable Identification & Role Assignment.
To benchmark this capability we introduce StatFormBench, built from five cross‑domain statistics textbooks and a data‑science case library. The benchmark covers diverse problem types, data representations, and scenario styles, containing 1,013 samples that span 20 coarse‑grained and 85 fine‑grained statistical problem categories.
Across 14 open‑ and closed‑source LLMs, the best zero‑shot models achieve only 72.0% fine‑grained classification accuracy and 63.2% variable‑set overlap. No model consistently excels on both subtasks, and enhanced prompting strategies yield only limited or inconsistent gains.
The benchmark data are released on Hugging Face (https://huggingface.co/datasets/THU-CongLab/StatFormBench) and the evaluation code on GitHub (https://github.com/THU-CongLab/StatFormBench).
Review