NeFut Logo NeFut
中 Admin Login

[CS.AI] Can Large Language Models Conduct Scientific Literature Reviews? A Benchmark of 102 Articles

Published at: 2026-11-05 22:00 Last updated: 2026-11-07 12:32
#AI #Machine Learning #LLM

This study evaluates six AI tools across four essential stages of a literature review: locating relevant papers, interpreting their content, synthesizing findings, and proposing future research. The benchmark derives from a published review of the Spanish plume weather pattern, comprising 102 source articles.\n\nIn the initial search, five tools collectively identified 57 papers, with the best single tool finding 42. Providing institutional library access raised Codex's discoveries from 42 to 82 papers and Claude Code's from 15 to 79.\n\nWhen the full set of 102 PDFs was available, three tools correctly recognized all 18 papers that substantially discuss the Spanish plume, yet mis‑classified 40% to 79% of the remaining 84 mentions as substantial.\n\nThe same three tools produced syntheses that were blindly evaluated by two domain experts. The syntheses were largely accurate but offered limited insight and occasionally contained unsupported claims, quotation errors, and exaggerated statements of consensus.\n\nAll three tools suggested research directions broader than the existing Spanish plume literature; only two approached questions identified in the benchmark review.\n\nNo single tool excelled across every task. Based on these findings, eight recommendations are offered for researchers employing AI tools in literature reviews, highlighting that LLMs can aid discovery and interpretation when institutional access is available, but cannot replace expert judgment in evidence evaluation or synthesis.\n\nReview

Original Source: https://arxiv.org/abs/2611.05459

[h] Back to Home