Decision‑making in the emergency department (ED) relies on heterogeneous data such as patient history, vital signs, lab results, and electrocardiograms (ECGs). Vision‑language models (VLMs) can ingest these modalities jointly, yet strong predictive performance does not guarantee that the model actually uses the patient‑specific ECG. We call this failure mode ECG Mirage: apparent multimodal capability without meaningful dependence on the correct ECG.
We distinguish two forms of ECG Mirage:
- ECG neglect – the ECG contributes little to prediction;
- ECG confusion – matched ECGs improve performance but the gap to mismatched ECGs is small.
To evaluate these behaviours, we keep the clinical text and prediction targets fixed and compare three settings: matched ECGs, outcome‑discordant mismatched ECGs, and no‑image inputs. Experiments on four VLMs using the MDS‑ED dataset target ICU admission and clinical deterioration. Matched ECGs provide no consistent advantage for either task.
We then train four restricted visual prompts via supervised learning followed by conditional direct preference optimisation, while freezing the VLM backbone. The tuned models achieve balanced accuracies of 70.6% for ICU admission and 67.5% for deterioration, widening the matched‑versus‑mismatched performance gap to roughly 16.5 and 5.5 percentage points, respectively.
In summary, this study identifies ECG Mirage in multimodal clinical prediction and introduces visual prompt tuning as an efficient mitigation strategy.
Review