Large language model (LLM) agents can keep scientific experiments running for months with minimal human supervision, yet reliable autonomous research also demands scientific judgment. Evaluating such judgment is challenging because research decisions are open‑ended and context‑dependent. To address this, we introduce SciΠvot, a benchmark comprising 248 cases derived from human corrections. The benchmark draws on six months of work in which senior researchers ran LLM agents on 26 projects across six scientific fields, repeatedly intervening to correct the agents' judgments. In each case the agent must choose between the original LLM plan and a human‑corrected alternative, using only the project record available before the correction as context. We compare LLM and human performance and examine the effects of reasoning effort, access to the project record, and reconsideration. Findings show that at medium reasoning effort with the project record, models systematically favor the original LLM plan over the human‑corrected option; four of five frontier models perform below chance. The best accuracy achieved in the project‑record condition across all models and effort levels is 63.5%, versus 73.4% for fresh researchers. At medium effort every model’s accuracy drops when the project record is provided, and reconsideration does not consistently improve overall performance. Even when a human‑corrected alternative is available, frontier models struggle with the selected research decisions, while SciΠvot enables systematic evaluation of scientific judgment for LLM agents.
Review