Sparse autoencoders (SAEs) have emerged as a promising technique for mechanistic interpretability by learning a set of sparse latent features that encode distinct concepts. However, vanilla SAEs struggle to learn modality-consistent concepts in vision-language models (VLMs), often resulting in fragmented coverage (i.e., disjoint regions) in the visual modality. To address this challenge, we propose a Structured Sparse AutoEncoder ($S^2AE$) that enforces concept consistency from both semantic and spatial perspectives in the visual modality.
Specifically, we group image patches based on Transformer attention similarity and spatial proximity, introducing a structured sparsity regularization during the training of the vanilla SAE. This regularization consists of exclusive sparsity for inter-group concept disentanglement and group sparsity for intra-group concept consistency, driving the latent neurons by SAEs to specialize in distinct, semantically grounded concepts.
Evaluated on the \texttt{Qwen2.5-VL-7B-Instruct} model, the method achieves a 6.06% average improvement in semantic alignment (mIoU) and 60.81 in representational efficiency (lower l0 norm) while maintaining near-perfect reconstruction fidelity with an Explained Variance above 99%. Cross-modal analysis further demonstrates that $S^2AE$ enhances neuronal monosemanticity by this visual structural prior, achieving a 3.08% average gain in semantic consistency and a 2.37% average gain in monosemanticity scores for both modalities of multimodal features, thereby fostering more coherent and disentangled representations.
Blogger's Review: This study successfully enhances concept consistency in vision-language models through structured sparsity regularization, showcasing the immense potential of sparse autoencoders in cross-modal learning. The structured approach offers new avenues for future multimodal learning exploration, warranting further investigation.