Explaining a vision‑language model (VLM) prediction requires tracing the influence of training examples back to the multimodal evidence that mediates it, yet most existing attribution methods operate only at the sample level. Recent latent‑mediated influence techniques provide feature‑level attribution for language models but lack the multimodal and spatial structure needed to ground influence in VLMs. To address this gap we introduce MOSAIC (Multimodal Orthogonalized Sparse Autoencoders for Influence Characterization). MOSAIC learns a shared sparse image‑text latent space whose concepts satisfy: (1) sparsity within each modality, (2) support on contiguous image regions, (3) alignment with the phrases that describe them, and (4) approximate decorrelation. We then apply EK‑FAC to decompose training‑example influence through these concepts and localize contributions to specific text phrases and image regions. We evaluate MOSAIC on Qwen and Gemma across captioning, chest X‑ray understanding, and visual question answering. Results show that the learned features are more localized and coherent without degrading task performance; ablating latents ranked by influence alters model outputs more than ablating latents ranked by activation or frequency. Moreover, visual evidence can substantially reorganize training‑data attribution while preserving more of the shared latent structure. MOSAIC thus connects where model behavior originates with what multimodal evidence carries the influence.
Review