Abstract
Open-weight medical language models are increasingly utilized in patient-facing and clinician-support applications. Their model cards prohibit specific behaviors—such as recommending exact drug dosages, issuing definitive diagnoses, prescribing treatments, adjudicating drug-drug interactions, and advising that emergency care can be skipped—yet a model card describes intended behavior, not robust behavior. We quantify that gap for MedGemma-4B under attacks that require no technical sophistication.
Methodology
We build a fully factorial benchmark of 5 guarded-behavior concepts, 50 deterministically templated questions, 6 lay-accessible attack manners, and 3 repetitions (4,500 generations). The model is served locally through Ollama under default sampling, and every response is coded as refuse/hedge/comply by three independent judges (an LLM judge, a transparent regex judge, and an NLI-entailment judge).
Results
Under the primary LLM judge, the overall Attack Success Rate (ASR, the fraction coded comply) is 38.0%. The two framings that reinterpret the request as legitimate dominate: recasting a question as a "medical board exam" item raises ASR from a 29.0% baseline to 53.1% (+24.1 points), and an appeal to an alleged doctor's authority raises it to 43.7% (+14.7). Crude instruction-override prefixes have no significant effect. Robustness is dominated by topic: the drug-interaction guardrail is nearly absent (83.2% ASR) while the emergency-deferral guardrail is strong (4.7%), and the authority framing is the only attack that breaches it.
Conclusion
Our findings motivate stronger deployment-time guardrails for open medical models.
Blogger's Review: This paper provides deep insights into the vulnerabilities of medical language models, particularly highlighting the shortcomings of existing safeguards, especially regarding drug interactions. Future research should focus on enhancing the robustness of these models in practical applications to ensure patient safety.