Abstract
Interpretability methods aim to reveal the features represented inside large language models (LLMs). Many existing methods begin with labeled examples of a human-defined concept that may reflect human biases, and then identify how that concept is represented within the model, for example in its activation space or through other decomposition methods.
We introduce \textit{Mining via Activation Geometry} (MAG), a simple unsupervised framework for extracting reasoning features from model activations by prepending the same natural-language instruction $Q$ to every input $p$, where $Q$ defines the reasoning feature of interest, such as "Can this object be found in the desert?" or "Is this prompt malicious?".
We measure how the instruction changes the model's internal representation using $m(Q \,|\, p) - m(p)$ at a single readout point. We explore eight different MAGs. The extracted reasoning features predict the models' own world understanding and judgment, can be approximated into a single activation direction. We found that some features are more linearly represented and some less, this linear representation, which is vector steering, can change the LLMs' decisions through activation steering by injecting reasoning features.
Finally, we use the same method to select the best training datasets for prompt-injection classifier probes: while similarity between ordinary activations is almost unrelated to downstream performance, RFD-based similarity achieves 94.7% Top-1 and 100% Top-2 accuracy.
Blogger's Review: This research delves into the unsupervised extraction of activation geometries in large language models, showcasing how instruction can guide the model's internal representations to enhance its understanding and judgment capabilities for specific tasks. This approach offers a fresh perspective on model interpretability, particularly in the context of training dataset selection, highlighting the potential of activation geometry in model optimization.