Federated Prompt Tuning (FPT) adapts large vision‑language models by freezing the pretrained backbone and optimizing only lightweight prompt parameters on each client, which reduces communication cost and parameter overhead. This design, however, leaves the shared immutable backbone as an overlooked attack surface. A malicious client can embed a backdoor in the representation space while keeping its uploaded prompt updates close to benign ones, making detection difficult.
ShadowFPT introduces a targeted backdoor attack. The attacker first pretrains a learnable shadow trigger against the frozen CLIP visual encoder, using either public auxiliary data or its own local data, so that triggered inputs are steered toward the target class in feature space. During federated prompt tuning, the malicious client fine‑tunes the trigger under the current global prompt and optimizes its local prompt on both clean and triggered samples. Only the prompt parameters are sent to the server, while the trigger and frozen encoders remain local.
By shifting most of the attack burden to trigger‑induced representation steering, ShadowFPT achieves high targeted misclassification while preserving stealthiness in prompt space. Experiments across multiple datasets, aggregation rules, and non‑IID partitions raise the attack success rate from 19.81% to 90.36% without degrading clean accuracy. The approach works for textual, visual, and joint vision‑language prompt tuning. These findings indicate that defenses relying solely on prompt‑update anomaly detection are insufficient against backdoors exploiting frozen backbones.
Review