Public crash databases are increasingly enabling automated safety analysis, yet translating crash‑severity prediction into public‑sector decision workflows remains challenging because models are typically evaluated only as ordinary classifiers. This work reframes dementia‑related crash severity modeling as a decision‑aware triage task: the system must assign each crash to no‑injury/property‑damage‑only (O), minor or moderate injury (BC), or fatal/severe injury (KA), while controlling outcome leakage, reporting severe under‑triage, calibrating confidence, and preserving every raw prediction for audit.
We use 4,781 Texas crash records that contain both structured fields and police narrative texts. The data are split stratified into 70% training, 15% validation, and 15% test. Baselines evaluated include pure structured models, pure narrative models, fusion models, calibrated fusion, BERT‑family models, and a locally deployed large language model (Gemma). Under leakage‑controlled conditions, Gemma achieves the highest observed macro‑F1 of 0.545 (95% bootstrap CI [0.507, 0.583]). The best calibrated fusion model reaches macro‑F1 of 0.522 with an expected calibration error (ECE) of 0.033.
Selective deferral improves performance on cases retained for automatic classification. At 70% coverage, macro‑F1 rises to 0.573 and the severity cost drops to 0.577, while the deferred 30% are treated as candidates for a proposed human‑review process and are not further evaluated in this experiment. The study contributes a reproducible, leakage‑controlled, and uncertainty‑aware evaluation framework for crash AI systems, emphasizing auditability and selective deferral rather than accuracy alone.
Review