Large Language Model (LLM) agents have shown promise for automating data‑science pipelines, yet their end‑to‑end performance hinges on the agent harness that defines tasks, tracks execution state, constrains outputs, and supplies evaluation feedback. Existing agents often keep this harness implicit, making results hard to reproduce, compare, and attribute across heterogeneous tasks.
DS‑Lighting introduces a unified harness toolkit that makes the harness explicit by decomposing it into four reusable layers: data, workflow, execution, and evaluation. Each layer offers a standardized interface, allowing diverse agents to be compiled into executable operator programs that support both fixed pipelines and adaptive search strategies.
The toolkit also integrates several open‑source data‑science benchmarks into an MLE‑Bench‑style task format, providing a shared task interface, sandboxed runtime, and metric protocol for controlled comparison.
Experiments across agents, harness configurations, models, and ablations demonstrate that explicit harness design improves reproducibility, comparability, and reliability while reducing avoidable system‑level failures in end‑to‑end data‑science workflows. The code is released at https://github.com/usail-hkust/dslighting.
Review