When we want conclusions about model representations that go beyond a single trained network, we must first define when two representations are considered equivalent. The Linear Representation Hypothesis is frequently invoked without specifying this equivalence. Different notions of equivalence preserve different structural aspects, so metrics, probes, and interventions that appear to study the same representation may actually correspond to distinct hypotheses.
We argue that the Linear Representation Hypothesis is not a single statement but a family of claims distinguished by the chosen equivalence relation. To formalize this, we use group actions: we specify the representation object (e.g., hidden‑layer vectors), the procedure that generates it (forward pass), and the property we wish to assert (e.g., linear separability), while incorporating the equivalences imposed by the model architecture.
This framework makes explicit how assumptions shift across metrics, read‑out points, and analysis stages. Using it, we audit common representation quantities—such as centered or normalized vectors—and recent interpretability studies, highlighting that hidden equivalence assumptions are often omitted and can lead to misinterpretation of results.
Review