This paper presents an LLM‑driven pipeline for extracting structured information from 536 peer‑reviewed agent‑based disease spread modeling papers. A schema covering model name, parameters, data sources, validation methods and other attributes was first defined, and multi‑turn prompts were crafted to guide GPT‑4.1 and GPT‑5.0 through the extraction process. The outputs were then benchmarked against a manually conducted systematic literature review (SLR), yielding paper‑level accuracies of about 77.95% for GPT‑4.1 and 81.67% for GPT‑5.0, while field‑level accuracies ranged from 32.40% to 100.00%, with lower performance on complex or subjective fields. A key observation is that agreement between the two LLMs serves as a potential quality indicator: low agreement often signals hallucinations, whereas high agreement combined with low overall accuracy may expose noise or errors in the human‑annotated dataset. The study offers practical prompt‑engineering guidelines and discusses both the promise and the limitations of scaling LLM‑based SLRs in the modeling and simulation domain.
Blogger's Review: The work convincingly demonstrates LLM feasibility for literature extraction and introduces a novel consistency metric, yet further advances are needed to handle nuanced, subjective fields reliably.