Automated research systems aim to automate literature synthesis, idea generation, experiment execution, manuscript writing, and peer review. Yet their evaluation is scattered across disparate tasks, benchmarks, and studies, making direct comparison difficult. This survey examines the literature from the perspective of evaluation design and evidence, focusing on six targets: literature synthesis, research ideation, executable workflows, scholarly writing and communication, automatic peer review, and end‑to‑end research.
We compare how tasks are constructed, what evidence sources are used, who the evaluators are, and how scoring is performed, to reveal which capabilities each design assesses. Three recurring lessons emerge: output checks, process checks, and human studies provide complementary information; evaluator calibration must be specific to the property being measured; and resource budgets together with attempt selection are essential for interpreting performance differences.
From these insights we outline diagnostic evaluation designs, highlight documented gaps in supporting evidence, and translate the comparisons into reporting and audit recommendations for specific evaluation settings. The survey assists readers in navigating existing evaluations, selecting appropriate benchmarks, and designing future studies.
Review