We performed a controlled ablation study on the SciFact benchmark, using only the live PubMed API as a tool. With the backbone model and output format fixed, we compared four harnesses: closed‑book answering, single‑shot retrieval using the claim as the query, a ReAct‑style agent that formulates its own PubMed queries, and an oracle supplied with gold abstracts, each paired with two efficient‑frontier backbones. Single‑shot retrieval finds the gold paper for merely 6% of claims and converts over‑confident errors into abstentions, offering no accuracy gain. Allowing the model to craft its own queries triples gold recall and yields the best verdict accuracy (62%), while the other realistic harnesses range from 49% to 59%; these differences are not statistically significant given our sample size, and agents consume less than half of their search budget. When provided with gold abstracts, the same models achieve 79%‑86% accuracy, a significant 17‑24‑point gap that identifies retrieval—not reasoning—as the primary bottleneck. A citation audit reveals that closed‑book models cite PubMed IDs that almost always exist (327/335) but are rarely relevant: a cross‑family LLM judge deems only 3 of those citations relevant, versus 67% relevance for citations generated by the search agent. Hence, existence checks alone are insufficient safeguards for scientific agents. Code, prompts, and all cached tool calls are released at https://github.com/saqlain2204/real-pmids-wrong-papers.
Review