In this paper, we explore the connection between an LLM's output distribution and the data used to train it. Specifically, we study the degree to which an LLM's next-token distribution aligns with the empirical next-token distribution (ENTD) given the context in the training data. The ENTD is an appealing target because it serves as the unrestricted global minimizer of the next-token cross-entropy loss used for pretraining and is an easily interpretable function of the pretraining corpus.
Our findings reveal that for a significant fraction of inputs, the LLM's distribution aligns almost perfectly with the ENTD, and the average agreement increases with model scale and training compute. However, there exists a long tail of input sequences where the LLM and ENTD differ significantly. We examine several possible sources of this discrepancy across the transformer architecture, training procedure, and finite-sample noise in the ENTD estimate itself.
More broadly, we hope our findings will inspire more work on 'data-centric mechanistic interpretability,' a complement to standard mechanistic interpretability that opens the black box of how model behaviors arise from the data, rather than how they are encoded in the learned weights.
Blogger's Review: This paper provides a deep analysis of the relationship between LLM outputs and training data, offering new insights into understanding model behavior. By investigating the empirical next-token distribution, the authors highlight performance differences in various contexts, laying a foundation for future model optimization and interpretability research.