Reproducible and analytically robust research findings are essential for scientific progress, yet assessing these properties usually demands costly manual effort. This work investigates whether large language model (LLM) agents can automate reproduction and robustness analysis, using the U.S. cancer registry SEER (Surveillance, Epidemiology, and End Results) as a medical case study.
We first introduce SEER‑Repro, a benchmark comprising 20 published SEER‑based papers together with the necessary data and resources to reproduce their results. Next, we develop an AI agent that attempts to reproduce 2,190 atomic claims, achieving a 77.4% success rate and providing executable baseline scripts for further investigation.
We then construct an agentic analysis pipeline that automatically identifies the analytical choices underlying each finding, proposes defensible alternatives, and reruns the analyses to see if conclusions change. Across fully reproduced findings, the pipeline flags potentially consequential analytical choices in 9.5% of cases, prompting expert review.
These results demonstrate that LLM agents can support scientific review by efficiently reproducing findings, systematically testing their robustness, and directing expert attention to potentially impactful analytical decisions.
Review