This paper explores a different failure mode in the context of model-specific adversarial attacks: the naturally occurring statistical signals in visual data that can behave like backdoor-like triggers without malicious insertion, termed as statistical adversaries.
We analyze the Imagenet dataset to identify patterns strongly associated with specific labels. Statistical controls are employed to eliminate random correlations from candidate signals. Ultimately, we demonstrate that these signals can directly and predictably alter model predictions.
These statistical adversaries are more targeted than generic corruptions and show transferability across different model architectures. This suggests that some vulnerabilities stem from the dataset structure and distribution rather than the peculiarities of individual models.
We conclude that ordinary datasets can harbor exploitable adversarial surfaces even without poisoning, indicating that dataset audits should treat spurious structures not only as sources of bias or interpretability failure, but also as latent attack surfaces for vision models.
Blogger's Review: This paper highlights the potential of statistical signals in vision datasets to become attack surfaces, urging caution in dataset usage. The vulnerabilities arising from dataset structure and distribution present critical implications for future research. A more comprehensive audit of datasets is necessary to address these latent security risks.