Telephone triage is a safety‑critical task: a nurse must decide from a brief intake note whether a caller needs an ambulance, an emergency department, an appointment, or self‑care at home. Directly querying large language models (LLMs) is unreliable—they tend to over‑escalate routine cases and cannot justify recommendations against an auditable clinical standard. ALLMTS addresses this by having the model execute the same commercially maintained telehealth guidelines that nurses use, rather than asking the model to triage on its own. The system first selects the guideline that matches the intake note, then evaluates the guideline’s triage assessment questions one by one. The disposition is determined deterministically by the first positive answer, following the exact traversal rule a nurse would apply. We evaluated ALLMTS on 566 synthetic, expert‑validated scenarios covering 14 dispositions and designed to test specific guideline questions. ALLMTS achieved 0.95 accuracy, compared with 0.47 for a raw‑LLM baseline. On a curated 50‑scenario open benchmark, ALLMTS reached 0.98 accuracy, on par with the 0.944 correct disposition rate recorded by a five‑nurse expert panel on the same cases. Crucially, baseline models exhibited over‑triage rates up to 0.50 (0.44 on the full set), whereas ALLMTS reduced over‑triage to at most 0.02 while keeping under‑triage low. These results support coupling LLM language understanding with executable clinical guidelines for reliable intake‑note triage.
Review