Information visualizations are widely used to communicate patterns, trends, and outliers. However, deceptive design choices—such as truncated or inverted axes, distorted aspect ratios, inappropriate encodings, and misleading color mappings—can systematically alter interpretation while preserving the underlying data.
As Vision-Language Models (VLMs) are increasingly used for chart understanding and analytical reasoning, assessing their robustness to such deceptive visualizations has become critical.
We introduce VisDeception, the first controlled paired benchmark for evaluating the robustness of VLMs to misleading chart designs. This benchmark contains 1,600 paired faithful and misleading charts spanning eight major categories of deceptive visualization tactics, where each misleading chart is paired with a faithful counterpart generated from the same underlying data.
To isolate deception-induced reasoning errors from baseline chart-understanding errors, we introduce the Deception Score, a paired evaluation metric that quantifies how misleading visualizations shift model responses away from the faithful interpretation of the data.
Across 32,000 responses from 10 state-of-the-art VLMs, we find that even advanced models remain highly vulnerable to deceptive visual manipulations. To improve robustness, we propose an inference-time multi-agent mitigation framework that grounds reasoning in structured chart metadata extracted from the visualization before answer generation, enabling models to reduce the influence of deceptive visual cues without requiring explicit user instructions.
Our findings reveal important reliability gaps in current chart-understanding systems and establish benchmark-driven evaluation, deception-aware metrics, and structured reasoning as promising directions for developing more trustworthy VLMs for visual analytics.
Blogger's Review: This study delves into the vulnerabilities of vision-language models when faced with poorly designed charts, presenting innovative evaluation and mitigation strategies. The introduction of the Deception Score and structured reasoning framework offers new insights for enhancing model reliability, warranting further exploration and validation in practical applications.