Vision‑language models (VLMs) are being deployed in high‑risk scenarios where a response that looks reasonable in general can become unsafe when the model lacks the user's medical, emotional or situational context. We introduce a benchmark for personalized safety, MPS‑Bench, containing 5,181 scenarios derived from 584 real images across 12 high‑risk domains, each paired with a hidden user profile. Evaluation of eight state‑of‑the‑art VLMs shows they almost always answer directly (86‑99%) instead of asking for missing context, and none scores above 2.6 out of 5 on personalized safety. An analysis of multimodal interaction reveals visual dominance: visual cues enter the text representation early and suppress textual risk signals during fusion, making late‑stage internal correction unreliable. Causal interventions demonstrate a two‑stage mechanism where visual affect first propagates into the text stream in early layers and then shapes the final decision via the altered text representation. Motivated by this insight, we propose PRISM, a lightweight input monitor that employs bidirectional cross‑modal modulation to predict when a query should be deferred. PRISM attains 0.978 AUC and strictly dominates the safety‑utility Pareto frontier for all tested models.
Review: The paper systematically exposes safety gaps of current VLMs under missing user context, pinpoints the root cause in visual dominance, and offers a practical deferral predictor, paving a clear path for safer multimodal systems.