LLMs are used for read‑and‑evaluate tasks in evidence synthesis such as study screening, data extraction, risk‑of‑bias assessment, and critical appraisal. These tasks vary in how much interpretation the model must perform: some only require retrieving simple facts like author name, sample size, or publication year, while others demand that the model read source material, apply predefined criteria, and render a judgment. Researchers need to verify that LLM judgments are accurate, backed by appropriate evidence, and reproducible under specified conditions. A correct judgment does not guarantee that the underlying evaluation process is reliable. To address this, we introduce TRAILS (Taxonomy of Reliability And Instability in LLM‑assisted evidence synthesis), which enumerates mechanisms that can compromise judgment reliability and guides the selection and reporting of suitable evaluation methods. TRAILS tackles two complementary questions: what mechanisms cause the same read‑and‑evaluate task to yield different judgments across repeated LLM runs, and what mechanisms can systematically affect the judgment or required output in ways that are not apparent from the judgment itself. We identify 38 mechanisms, describe the deployment settings where each can operate, propose testing approaches, and suggest mitigation strategies. Our findings show that repeatability or accuracy alone cannot rule out hidden judgment failures; therefore human evaluation should focus on whether a judgment is grounded in the supplied evidence, free from inappropriate external influence, and produced through a process that can be examined and verified.
Review