Frontier coding agents are increasingly framed as end‑to‑end automated researchers, and ML for science offers a demanding test: improving a scientific model often requires domain insight beyond routine ML engineering. We introduce Agent4Sci, a long‑horizon testbed that measures how far today’s agents can improve the published best models on scientific ML benchmarks, spanning 20 benchmarks across 11 domains.
Instead of a leaderboard, Agent4Sci asks what kinds of progress agents can make under in‑the‑wild conditions and where they fall short. Empirically, today’s frontier agents advance the published best on every benchmark, with Claude Opus achieving a 10.4% mean relative improvement and no run flagged for academic misconduct. These gains come almost entirely from a bag of ML‑engineering tricks rather than research‑level insight.
Trajectory analysis surfaces three recurring limitations: novelty miscalibration, resource under‑utilization, and rapid collapse into local engineering sweeps. Adding lightweight plug‑in skills for research ideation, experiment management, and scheduled checklists raises Claude Opus from 10.4% to 20.1% and Codex from 4.4% to 8.1%, while yielding more diverse, domain‑relevant interventions.
Agent4Sci gives the community a way to track and push frontier agents from pure engineering toward domain insight on real‑world ML‑for‑science problems. Code and data are available on GitHub (https://github.com/maszhongming/agent4sci) and Hugging Face (https://huggingface.co/datasets/MingZhong/agent4sci).
Review