Abstract
As advanced agents increasingly demonstrate the potential to operate as autonomous engineers, there is a growing demand for evaluation benchmarks that capture the complexity of real-world development. Existing benchmarks typically evaluate code-centric or data-centric capabilities in isolation, leaving a clear gap with real development scenarios.
In this paper, we introduce CODA-BENCH, the first benchmark to jointly evaluate code and data intelligence in a data-intensive environment. We construct a data-intensive Linux sandbox based on the Kaggle ecosystem, containing hundreds of datasets, where agents must actively explore complex file hierarchies to identify relevant resources and generate code for data-driven analytical tasks.
CODA-BENCH comprises 1,009 tasks spanning 31 communities, with each task environment containing an average of 980 files, simulating realistic data scale and noise.
Evaluations of advanced agents reveal that even top-performing systems struggle to effectively integrate data discovery with code execution, achieving a success rate of only 61.1%. These results highlight a substantial gap in current agentic capabilities for data-intensive tasks and point to promising directions for future research.
Blogger's Review: The introduction of CODA-BENCH highlights the need for evaluating agents in complex data environments, showcasing the challenges of integrating code and data intelligence. Future research should focus on enhancing agents' capabilities in real-world data handling to meet the growing demands.