We introduce Astro2Method, a benchmark that evaluates how large language models (LLMs) select machine‑learning methods for astronomical problems. The benchmark comprises 500 expert‑reviewed binary questions drawn from 346 astronomy papers published between Jan 2024 and Jun 2026. Questions are split into two categories: Easy, where authors report clear performance gaps and provide full explanations, and Hard, where only partial explanations or experimental descriptions are available.
Direct prompting of five popular LLMs yields 87.50%‑92.55% accuracy on Easy questions but only 58.87%‑64.52% on Hard ones. We also tested retrieval‑augmented generation (RAG) using historical papers and several agent frameworks that add reasoning or retrieval steps (e.g., ReAct, Self‑Ask, CRITIC). None of these approaches consistently improved performance.
Agents that performed web searches helped only when they actually retrieved the original paper underlying the question; otherwise, no systematic gain was observed. A separate experiment masked the method names in the source paper’s pipeline description while keeping the experimental details. This masking dramatically boosted Hard‑question accuracy, matching or surpassing the benefit of web search.
Overall, the results suggest that concrete experimental information is key to distinguishing ML methods, yet extracting or inferring such details from related studies and domain knowledge remains a challenge for LLMs in astronomy.
Review