NeFut Logo NeFut
Admin Login

[CS.AI] DSAgentBench: Evaluating Agents in Real-World Data Science Workflows

Published at: 2026-08-12 22:00 Last updated: 2026-08-13 01:53
#AI #Machine Learning #Data Structure

A recent research paper introduces DSAgentBench, a benchmark for evaluating whether agents can automate end-to-end data science workflows. Real-world data science involves long-horizon workflows that span data wrangling, exploration, modeling, visualization, and validation, and require coordinated use of tools such as notebooks, IDEs, terminals, browsers, and databases within real operating environments. DSAgentBench contains 275 diverse tasks covering the entire data-science life-cycle, reflecting the complexity and tool coordination required in practice. Each task requires grounding decisions in intermediate outputs and coordinated tool use, and includes a deterministic evaluator that verifies analytical correctness, visual outputs, and model performance rather than code-only execution. Our extensive experiments with 15 closed- and open-source models show that even the strongest agent, Claude-4.6-Sonnet, achieves only 56.70% task success, while all open-source agents remain below 1%, frequently failing at tool orchestration, OS grounding, and multi-step reasoning. These results reveal a substantial capability gap between current agentic systems and real data-science workflows, positioning DSAgentBench as a foundation for developing grounded, verifiable, autonomous data-science agents. Blogger's Review: DSAgentBench provides an important benchmark for evaluating the capabilities of agents in real-world data science workflows, highlighting the limitations of current agent systems and the direction for future development.

Original Source: https://arxiv.org/abs/2608.10366

[h] Back to Home