Abstract
As part of a series on first-principles modeling of cognitive functions, this paper attempts to provide a mathematical formulation of thinking and perception. It formally derives slow thinking or more generally, active perception, encompassing the design, training, and inference of slow thinking large language models.
Our starting point is the lifting and projection of probability distributions on observable and latent spaces, aiming to represent complex data distributions by simple function families such as neural networks. A theory called "active lifting" is proposed, based on the sampling of latent sequences and an intrinsic drive to reduce uncertainty at maximum rate. It derives a large design space containing the slow thinking models in a subspace we call the static theory. These models are positioned on the representation and sampler hierarchies induced by the static theory and can be upgraded by climbing these hierarchies.
Active lifting further derives an inference process with an internal time axis, and a training objective resembling minimum-length coding as well as the invention of languages. Thus, it characterizes the agency of perception, including the emergence of slow thinking formats. Technical by-products of this theory include a three-stage pathway for improving slow thinking models, a unified approach to constructing encoders and generative models for all data modalities, a priori formation of human-like visual representations, and a potential solution to policy collapse.
Blogger's Review: This theory provides a fresh perspective on understanding human cognition by mathematically modeling slow thinking and active perception, revealing deep intrinsic mechanisms with significant application potential, especially in building smarter AI systems. Its multi-layered design and training methods could greatly advance the progress of large language models.