The reviewer lottery error refers to mistakes that arise when only a small set of reviewers address a subset of a paper's issues, while a larger pool of eligible reviewers could raise many more concerns. We propose mitigating this error by surfacing a checklist of criticisms learned from a historical review corpus. Our method first builds a tree of checklists that represent successive coarsenings of the atomic review‑claim space, then selects the coarsening that maximizes a theoretically motivated measure of correlated agreement among reviewers. We implement this with an LLM‑assisted pipeline and evaluate it on NeurIPS 2021 and ICLR 2022 review data. In simulated reviews using LLM reviewers, the learned checklist reduces the mean squared error (MSE) of a three‑reviewer committee by $58\%$ on NeurIPS and $21\%$ on ICLR, and in both venues the checklist with the highest correlated agreement yields the lowest error for committees of three or more. In a human study with 20 participants, the NeurIPS checklist lowers the MSE of paper ranking (against the official NeurIPS 2021 order) from $0.48$ to $0.22$.
Review