A language model builds its next‑token prediction layer by layer, and lens methods track this evolution by decoding intermediate hidden states into tokens. The decoding depends both on the hidden state and on the readout matrix (the unembedding matrix). Most lenses are fitted on a specific corpus; experiments show that swapping the fitting corpus while keeping the model fixed can yield different token predictions, a phenomenon we call corpus conditionality.
To study readout structure without corpus influence, we introduce Sparse Readout Prism (SRP). SRP uses only the weights of the readout matrix, decomposes it into sparse features, and can express any token logit or logit difference as a sum of contributions from these sparse features.
With this representation, readout features become a new elementary unit for lens interpretation, exposing structure hidden by token identities and enabling comparisons across tokens, contexts, layers, and different lenses. Replacing the original readout with SRP’s sparse approximation reconstructs an additional 8.9%‑17.3% of tested logit differences compared with six baselines based on geometric relations among readout rows. Ablating features shifts logit differences proportionally to their SRP contributions. Although token readings vary with the fitting corpus, the dominant readout feature remains stable. Because SRP is built without any corpus, it provides a corpus‑independent control for lens analyses.
Review