What does a language model predict when it receives little evidence? The answer lies in the geometry of its unembedding matrix. A single direction in this matrix encodes the unigram distribution of the training corpus, acting as the Bayesian prior the model falls back on under uncertainty. We call this the “direction of ignorance”. The phenomenon appears in four model families—Llama, Qwen, Gemma, and Pythia—spanning 0.4 B to 405 B parameters. Projecting the final prediction state onto this direction yields a per‑token prior loading factor $\lambda$. Empirically $\lambda$ decreases steadily as the context becomes more informative. Mathematically the projection decomposes the prediction state into two orthogonal vectors: a unigram prior raised to the power $\lambda$ and a context‑driven likelihood, exactly matching the two factors of a tempered Bayesian update. This geometric‑probabilistic view calibrates $\lambda$ across model sizes and families; larger models rely less on the prior in the high‑context limit. Finally, causal experiments show that raising or lowering $\lambda$ at the prediction state steers the output toward or away from the unigram prior in KL divergence.
Review: The paper uncovers a concrete probabilistic geometry inside LLMs, offering a measurable explanation for behavior under scarce cues and a direct knob to modulate prior reliance.