Automated reviewing systems are increasingly judged by the quality of their reviews, yet a review only matters when it leads to measurable paper improvements. We introduce AppliedScientist, a closed‑loop framework that couples an autonomous AI scientist with an AI reviewer and evaluates it by iteratively revising rejected papers across diverse subfields.
During revision the AI scientist can consult its own previous drafts, mimicking how human authors build on earlier versions; the reviewer generates feedback independently and has no memory of prior comments or scores, thus avoiding bias.
We compare three revision settings: initialization with original venue reviews, initialization with AI‑generated reviews, and fully autonomous self‑revision using the same fixed prompt each round. To assess the reviewer’s guidance we also employ Stanford Reviewer as an independent evaluator for human‑initialized revisions.
Results show reviewer‑guided revision consistently outperforms fixed‑prompt self‑revision, and Stanford Reviewer assigns higher scores to later revisions. AppliedScientist resolves 128 of 150 (85.3%) execution‑related weaknesses but only 2 of 18 (11.1%) idea‑related weaknesses, indicating that iterative revision effectively improves experiments and implementation while rarely addressing novelty or significance concerns.
This study demonstrates the promise of a closed‑loop AI review‑revision pipeline and points toward more automated scientific workflows.
Review