The rapid expansion of scientific publishing has put increasing strain on peer review, especially in machine‑learning venues, leading to concerns about declining review quality and growing reviewer workload. Large language models (LLMs) have been proposed as automated review assistants, yet existing evaluations mainly measure how well they imitate human‑written reviews, overlooking the core functions of peer review.
We adopt a verification‑centric perspective, treating error detection as a critical, resource‑intensive task. To this end we build a scalable benchmark by synthetically inserting logical contradictions into conference papers, creating unambiguous targets that enable systematic comparison of review systems’ detection abilities.
We also introduce a Multi‑Layered Review (MLR) framework that first forces the model to achieve deep manuscript comprehension before generating review comments. This mirrors human reviewing practice and improves token efficiency.
Across experiments, our approach aligns closely with human review scores, achieves high error‑detection performance, and offers complementary insights into reviewer focus. The gains stem from both the choice of underlying LLM and the design of the system. At the same time, we confirm persistent vulnerabilities to adversarial manipulation, highlighting the need for robustness in automated review tools.
Our findings underscore the importance of rigorous, error‑focused evaluation to guide the responsible deployment of LLM‑based tools in peer review and other critical scientific workflows.
Review