Reproducibility checks during peer review require a clear definition of what the authors' code should reproduce. We introduce an author‑in‑the‑loop collaborative audit that treats author declarations as explicit inputs for an AI agent’s investigation and assessment. Authors fill a structured checklist describing reproduction instructions and known limitations. A two‑stage workflow on GitHub then connects the agent’s automated checks to author clarification and revision, followed by a re‑assessment by the agent. The agent independently decides whether the results have been reproduced and whether the authors need to provide clarification or make repairs, while human reviewers approve the agent’s requests and determine if concerns are resolved. In a conference deployment we applied the system to 36 papers covering 133 experimental tables and figures, identifying 20 issues after human review. Subsequent checks confirmed that all issues were addressed, although full reproduction remained unverified for some experiments. We also compared repeated audits using complete versus blank checklists, showing that author declarations shape the agent’s investigative path and assessment outcomes. Our contribution is a collaborative workflow and agent design that leverage author knowledge to guide reproducibility auditing, with empirical evidence that author‑agent‑reviewer collaboration can effectively surface and fix reproducibility problems.
Review