Electronic health records combine free-text clinical narratives with structured measurements such as vital signs, laboratory values, and comorbidities. However, most clinical prediction systems still rely on task-specific fusion architectures, pairing dedicated encoders for each modality with learned combination mechanisms that must be re-engineered for every new task and clinical setting. We propose a simpler alternative: convert all patient data, regardless of modality, into a single natural language sequence and fine-tune a pretrained language model end-to-end, with no architectural modification for fusion.
We evaluate this approach across three clinically distinct prediction tasks: in-hospital mortality on MIMIC-III, graft failure prediction using longitudinal data from a German transplant center, and emergency triage classification from ambulance records—comparing encoder-based (ModernBERT) and decoder-based (Llama 3.1, Gemma, DeepSeek-R1-Qwen, Qwen3) fine-tuning against established multimodal baselines and, for graft failure, a gradient boosting model currently used in clinical practice for post-transplant patient management.
Across all three tasks, unified textual serialization matches or exceeds task-specific multimodal baselines, and outperforms the clinically deployed gradient boosting system on graft failure prediction. These results indicate that a single serialization-based paradigm, without bespoke fusion architectures, is sufficient for multimodal clinical prediction—substantially reducing system complexity while matching or exceeding specialized designs.
Blogger's Review: This study demonstrates the powerful capability of large language models in clinical prediction by unifying different modalities into textual sequences. This approach not only simplifies system architecture but also outperforms traditional multimodal models across multiple tasks, showcasing its potential application in the medical field, warranting further exploration.