Large language models (LLMs) are often evaluated under the assumption that perfect reliability is achievable given sufficient scale. However, we show that this assumption is information-theoretically unjustified. Every generative task has a reliability ceiling that no model can exceed, determined by how much output uncertainty can be resolved from observable context.
This gap decomposes into a resolvable component that can be closed with additional context and a subjective component that is inherent to task ambiguity. Autoregressive generation further degrades this ceiling at a rate governed by the task's dependency kernel, which quantifies inter-token correlations in the output.
From these two primitives, we derive a first-principles scaling law where LLM performance is bottlenecked by the scarcer resource: training data or model capacity. This law recovers the Chinchilla scaling law as a special case and provides a structural account of when scaling improves reliability.
Beyond scaling, our framework unifies diverse practical phenomena, such as the benefits of retrieval-augmentation and the spectral mechanics of catastrophic forgetting. Our work formalizes the resource-complexity tradeoffs that govern model performance across domains, offering a unified theory of performance limits in generative language models.
Blogger's Review: This paper delves into the theoretical limits of large language models, revealing the intricate relationship between reliability and task ambiguity. By introducing an information-theoretic perspective, the authors provide a new framework for understanding model performance, which is significant for future model design and optimization.