AlphaIdeaBench defines 100 research‑idea generation tasks, split evenly between workshop calls for papers and open problems gathered from surveys. Each task includes an explicit compute, API credit, and time budget and follows a shared proposal schema. An LLM‑based judge evaluates proposals on 12 dimensions covering validity, novelty, and feasibility.
To validate the judge, we compare its scores on four novelty axes—problem, method, scenario, and insight—with scores derived from reviewer comments on 600 papers from ICLR 2026, NeurIPS 2025, and ICML 2025. Gemini 3.7 Flash stays within one point of the reviewer‑derived scores for problem and method novelty on 94%–97% of papers, while scenario novelty shows lower agreement.
We benchmark eight published ideation harnesses on a common backbone and, within one harness, eight different backbone LLMs. CoI‑Agent achieves the highest overall score, narrowly beating SciMON, which leads on method and insight novelty. All harnesses score low on scenario novelty and rarely alter the original problem setting. The backbone largely determines which open problem a proposal targets, whereas the harness decides whether a proposal tackles a single problem or combines several. Weaker backbones such as Llama 4 Scout fail to execute the harness strategy and receive low feasibility scores.
When varying only the budget on three topics, larger budgets change the experimental setup for compute‑intensive topics but add merely extra tests for a cheap one. We release the tasks, rubrics, and scores to enable standardized, reproducible benchmarking of future research‑ideation agents under constraints.
Review