Sparse autoencoders (SAEs) explain foundation‑model representations by decomposing activations into sparse combinations of learned dictionary atoms. Yet the dictionaries learned by different training runs often diverge, limiting the reliability and reproducibility of SAE‑based analyses. To address this, we introduce Consistent Sparse Dictionary Learning (ConSDL), which replaces the standard linear encoder with per‑sample sparse coding.
A synthetic analysis shows that coefficients obtained from a single linear‑encoder pass are suboptimal compared with those from per‑sample optimization, indicating that sparse coding provides a more reliable mechanism for dictionary learning. Across 16 synthetic configurations, ConSDL achieves near‑perfect recovery of both dictionary atoms and coefficients.
When transferred to real‑world vision and language benchmarks, ConSDL attains state‑of‑the‑art cross‑seed consistency: dictionary consistency improves by 17.6% and coefficient consistency by 35.9% over the average of six SAE baselines.
Review