We assess whether AI agents can reproduce the reported findings of published biomedical studies using publicly available data. Prior evaluations often conflate a model's prior knowledge with retrieval of the published answer. To address this, we introduce BioStudyBench, a benchmark comprising 25 long‑horizon analysis tasks drawn from studies first published between July and September 2026—after the knowledge cutoffs of the evaluated models. The tasks were semi‑automatically filtered from 404,019 PubMed records. For each task the agent receives a neutral research question but no data files; it must locate and download the relevant public data, search the literature via tools that return only records dated before its cutoff, and report findings through data analysis. To isolate the benefit of data and tools, we run every task both with and without access to them. Across eight models, access to data and tools raises the pass rate by an average of 47 percentage points over the no‑data baseline. Open‑weight models lag behind closed‑weight ones at all scales, with the best open‑weight model achieving 81.3% pass rate versus 94.7% for the best closed‑weight model.
Review