Abstract
Auditing a new language model usually means relearning and reinterpreting its internal features from scratch. We propose a reference feature atlas: a sparse feature library trained once on a reference panel and reused for new targets, which attach by fitting only a linear decoder. This yields two complementary views.
Atlas Channel
The atlas channel reads the target on already interpreted panel features, providing a stable coordinate system across models.
Residual Channel
The residual channel learns features only from what the atlas fails to reconstruct, making "outside the reference panel" an explicit audit signal.
We train leave-one-out atlases over five 7-9B instruction-tuned models and audit held-out Mistral and Qwen targets. On three controlled LoRA hidden objectives injected into both targets, the residual channel makes the planted mechanism perfectly controllable at runtime while matched controls stay unaffected and recovers the planted objective as the top-ranked latent across both targets; on Mistral, where the per-target SAE and pairwise crosscoder baselines are retrained for a head-to-head benchmark, both baselines fail to do so. On Qwen-2.5, the same channel additionally reveals a panel-relative political-framing cluster; steering it shifts the audited framing metrics while out-of-domain controls remain unchanged.
Blogger's Review: This research introduces a highly efficient and reproducible method for auditing language models through the use of reference feature atlases. By leveraging linear decoders and sparse feature libraries, the authors successfully demonstrate how to maintain consistency across different models while revealing potential biases. The controllability and stability of this approach provide significant insights for future model audits.