As large language models (LLMs) are increasingly deployed in online medical consultations, existing benchmarks are poorly aligned with real clinical practices. Many rely on synthetic conversations or patient simulators, neglect patient-uploaded medical images, or evaluate open-ended clinical responses using multiple-choice or lexical-overlap metrics that fail to reflect clinical quality effectively.
We introduce MedRealMM, a large-scale benchmark for multimodal online medical consultation built from de-identified patient-doctor interactions collected from a nationwide Chinese internet hospital. MedRealMM employs a Multimodal Clinical Challenge Point (MCCP) extraction framework to identify clinically demanding moments in authentic consultation trajectories, converting each into a standardized next-response generation task while preserving the preceding text-image context.
Each instance is paired with a case-specific rubric refined by physicians that rewards clinically desirable behaviors and penalizes unsafe, unsupported, or contradictory responses. The current release contains 5,620 real-world multimodal cases spanning 64 clinical departments. We evaluate 19 general-purpose and medical-specialized LLMs, including text-only and multimodal systems. Our results indicate that image information is critical for reliable clinical performance, and that current frontier models remain below the online physician response.
Although some frontier models satisfy as many or more positive clinical criteria than physicians, they trigger more negative criteria, indicating that safety-sensitive error avoidance remains a central bottleneck. MedRealMM provides a realistic and reproducible benchmark for evaluating multimodal medical reasoning in real-world online consultations. The dataset will be publicly available on Hugging Face at Hugging Face.
Blogger's Review: The launch of MedRealMM offers a pragmatic evaluation tool for online medical consultations, particularly in processing multimodal information. While current models show promise in certain metrics, the focus on safety issues remains critical. Future research could pursue more efficient error avoidance mechanisms.