Large language models have dramatically lowered the barrier to writing code, giving rise to data‑science agents that can automate the entire pipeline from task formulation to model development. Existing evaluations focus on downstream predictive performance and code correctness, overlooking the core principles of data science—methodological rigor and domain knowledge. Consequently, relying solely on accuracy fails to expose issues such as data leakage or improper validation, hindering the development of robust agentic systems.
This paper evaluates agentic data science beyond accuracy from three angles:
- A large‑scale analysis of failure modes across 220 data‑science tasks for popular agentic frameworks and models.
- A reusable, extensible test suite that curates tasks and deliberately probes six typical failures (e.g., data leakage, incorrect cross‑validation) in a controlled setting.
- A qualitative assessment that compares agentic pipelines against expert solutions from Kaggle competitions, examining problem‑solving creativity and domain understanding.
Key findings:
- More than 70% of agentic solutions exhibit at least one failure mode, and increasing model capacity or framework complexity does not substantially reduce this rate.
- The occurrence of data‑leakage failures correlates positively with predictive performance, indicating that higher scores often accompany methodological flaws.
- Compared with competition‑winning data scientists, agents generally lack creative problem solving and deep domain expertise.
We argue that principled failure‑mode analysis should become an integral part of agentic data‑science evaluation to foster trustworthy pipelines.
Review