NeFut Logo NeFut
中 Admin Login

[CS.AI] Visual-Invariance-Augmented Feature Optimal Alignment for Transferable Adversarial Attacks against Closed-Source MLLMs

Published at: 2026-10-09 22:00 Last updated: 2026-10-12 00:49
#AI #Machine Learning #LLM

Multimodal large language models (MLLMs) remain vulnerable to transferable adversarial examples in black‑box settings where only open‑source surrogate models are available. Existing targeted transfer attacks mainly align adversarial and target samples using global image‑level features such as the encoder’s [CLS] embedding. This coarse alignment neglects patch‑level visual structures, limiting transferability across heterogeneous closed‑source MLLMs.\ \ We introduce IAU‑FOA (Visual‑Invariance‑Augmented Feature Optimal Alignment), which improves targeted transfer attacks by aligning samples at both global and local levels. A cosine‑based loss narrows the global semantic gap, while patch tokens are clustered into compact local patterns and matched via optimal transport for fine‑grained feature alignment.\ \ Balanced optimal transport enforces fixed marginal masses even for local clusters lacking reliable counterparts, potentially generating misleading alignment gradients. To mitigate this, we propose confidence‑adaptive unbalanced transport, which relaxes mass constraints for weakly matched clusters, reducing unreliable local alignment and enhancing overall transferability.\ \ We also investigate input transformations and propose visual‑invariance augmentation: bidirectional pixel‑intensity rescaling combined with per‑channel white‑balance adjustment to emulate variations in exposure, contrast, illumination, and color temperature. This encourages adversarial perturbations to generalize across different visual encoders.\ \ Extensive experiments on both open‑source and closed‑source MLLMs demonstrate that IAU‑FOA consistently outperforms state‑of‑the‑art transferable attack methods in success rate and efficiency. The code is released at https://github.com/jiaxiaojunQAQ/IAU-FOA.\ \ Review

Original Source: https://arxiv.org/abs/2610.06977

[h] Back to Home