We introduce PRISM (Population‑Retrieved Inference for Structured Multimodal Prognosis), a multimodal language model designed for clinical prognosis. At the feature level, each structured measurement is tokenized numerically; longitudinal ECGs are aggregated with a recency‑weighted scheme to capture the most recent physiological signals; and a dual‑modality population memory bank stores embeddings of both structured features and waveforms. Retrieval operates in representation space: the model receives mean‑pooled embeddings of the most similar training‑cohort encounters, without inserting any individual record into the prompt.
On the 1,443‑task MDS‑ED benchmark, PRISM achieves AUROCs of 86.2% (diagnosis), 93.1% (deterioration), 91.8% (ICU admission) and 92.8% (mortality). These results leverage a 96‑hour pre‑arrival window unavailable to reported baselines. When restricted to the benchmark‑window inputs only, PRISM reaches 0.8895 AUROC, slightly below UniPACT (0.8937) but above the original MDS‑ED (0.8889).
Ablation studies show that a compact fusion model with the same five evidence streams and frozen encoders attains 0.7652 AUROC, far lower than PRISM’s 0.9092. Replacing numerical EHR encoding with text serialization drops AUROC by 3.66 points; removing pre‑arrival history reduces it by 1.97 points; and zeroing out retrieval lowers it by 0.99 points. Genuine retrieval outperforms random‑neighbor retrieval by 0.97 macro‑average points, although a paired task‑level sign test does not reach significance (p=0.067).
On the external MC‑MED cohort, supervised fine‑tuning yields an AUROC of 0.8611, surpassing locally trained gradient boosting (0.8227) and source‑only PRISM (0.7841), supporting the value of structure‑aware multimodal integration and supervised external adaptation.
Review