As Large Language Models (LLMs) are increasingly utilized in daily operations, white-box monitoring is being adopted as an auditing tool to ensure safe model behavior. However, white-box monitors can be circumvented, and the mechanisms behind such evasion have not been systematically characterized, nor have principled defenses been proposed. This study addresses both challenges.
Controlled red-team experiments reveal two primary evasion strategies: geometric shifting, defined as the systematic migration of information between linear and non-linear representational subspaces, and covariance manipulation. These mechanisms account for the failure of single-detector approaches, as information migrates to subspaces inaccessible to individual detectors.
This issue is urgent due to growing evidence that models are becoming evaluation-aware, enabling misaligned objectives to exploit these vulnerabilities and evade monitoring during deployment. In response, SafetyNet is introduced as a principled ensemble, serving dual purposes: it provides further empirical validation that our mechanistic findings are real and actionable, and it offers a concrete starting point for future work on robust latent-space monitoring.
The study experiments across five model families on the MAD and Anthropic Sleeper Agent benchmark, with SafetyNet achieving around 100% AUROC scores, outscoring Beatrix and CROW. The code is available at: GitHub.
Blogger's Review: This paper rigorously reveals potential vulnerabilities in white-box monitoring through well-designed experiments and proposes effective defense strategies. The introduction of SafetyNet provides a new perspective for future monitoring research, showcasing the powerful capabilities of latent-space monitoring, making it worthy of further exploration and application.