Abstract
Large language models (LLMs) exhibit a remarkable ability to predict next observations from Hidden Markov Models (HMMs) via in-context learning (ICL), yet the underlying algorithm remains undetermined. Previous work has proposed various candidates without consensus, and none has been grounded in the model's internal activations. We address this gap with a three-stage pipeline.
Stage One
We empirically compare LLM behavior against a suite of candidate algorithms, narrowing the space to three classes—though no single class explains LLM behavior across all HMM settings and sequence lengths.
Stage Two
We derive theoretical connections between the three classes and demonstrate how each can be implemented in-context by a Transformer, validating this construction in a small trained Transformer.
Stage Three
Returning to pre-trained LLMs, we introduce the Principal Activations Probe (PAP), a layer-wise probing and intervention method that isolates algorithmic signals in model activations. PAP reveals low-dimensional linear representations that causally drive model predictions and track empirical ICL performance. It further shows how these representations shift with properties of the underlying HMM regime; distinct computational stages are localized to different layers.
Together, our results connect the in-context behavior of pre-trained LLMs to the underlying internal mechanisms and enhance our understanding of how LLMs perform ICL on HMMs.
Blogger's Review: This paper offers a systematic experimental and theoretical analysis of the internal mechanisms of LLMs when dealing with HMMs, particularly through the PAP method that uncovers algorithmic signals in model activations. This research not only fills a gap in the existing literature but also provides a fresh perspective on understanding the behavior of large language models.