This study explores how fine-tuned Large Language Models (LLMs) can covertly encode prompt secrets via steganographic channels. Previous works demonstrated this threat but relied on trivially recoverable encodings. We formalize payload recoverability through classifier accuracy, noting that previous schemes achieved 100% recoverability. In response, we introduce low-recoverability steganography, replacing arbitrary mappings with embedding-space-derived ones.
For Llama-8B (LoRA) and Ministral-8B (LoRA) trained on TrojanStego prompts, exact secret recovery increases from 17% to 30% (+78%) and from 24% to 43% (+80%), respectively. For Llama-70B (LoRA) trained on Wiki prompts, recovery climbs from 9% to 19% (+123%), all while reducing payload recoverability.
We then discuss detection, arguing that detecting fine-tuning-based steganographic attacks requires methods beyond traditional steganalysis. Standard approaches measure distributional shifts, which are expected side effects of fine-tuning. Instead, we propose a mechanistic interpretability approach: linear probes trained on later-layer activations can detect secrets with up to 33% higher accuracy in fine-tuned models compared to base models, even for low-recoverability schemes. This suggests that malicious fine-tuning leaves actionable internal signatures suitable for interpretability-based defenses.
Blogger's Review: This paper introduces an innovative low-recoverability steganography scheme that significantly enhances the concealment of steganographic information. Moreover, using mechanistic interpretability for detection provides a fresh perspective on defending against fine-tuning attacks, showcasing the potential of internal model features. This research offers valuable insights into LLM security and warrants further exploration.