As Video Large Language Models (LLMs) are increasingly deployed in real-world applications, ensuring their safety alignment has become critical. Counterintuitively, harmful videos paired with benign queries achieve higher attack success rates than the same videos paired with explicitly harmful queries.
To understand this vulnerability, we present V-DEAL, a three-level diagnostic framework that jointly analyzes this failure across model behavior, understanding, and internal representations. By progressively ruling out perception failures and quantifying the model's internal refusal tendency, V-DEAL provides a new diagnostic perspective for analyzing the underlying mechanism of the observed vulnerability.
We tested six Video LLMs on three public benchmarks and found that models correctly recognize harmful video content with over 81% accuracy, yet the average attack success rate still reaches 48.33% when pairing harmful videos with benign queries. Hidden-state analysis further shows that visual understanding activates a weaker refusal tendency than textual understanding.
Furthermore, we introduce a prompt injection intervention method that reduces attack success rates by an average of 48.24 percentage points and achieves performance comparable to prior fine-tuning-based methods, providing an effective and practical means to address such safety risks in Video LLMs.
Blogger's Review: The introduction of the V-DEAL framework offers a fresh perspective on the safety of Video Large Language Models, particularly by analyzing the internal refusal tendencies of models. This approach not only reveals potential risks when handling benign queries but also provides effective intervention methods, making it worthy of exploration and validation in broader applications.