Abstract
Existing hypotheses represent a concept in a large language model (LLM) as a single point, a linear direction, or a Gaussian cluster, yet it remains unclear how and why such structures emerge. Here, we show that concept geometry can be precisely characterized via Laguerre Geometry, in which a concept is defined as a region—a Laguerre-Voronoi cell or a union of cells—allowing us to strictly define, measure, and separate concepts.
Building on this formulation, we show that finer-grained concept structures, such as inclusion and hierarchy, are naturally revealed by the Laguerre weights. We then push this geometry inside the transformer. Decomposing each layer into piecewise-linear operators, we show that a token's hidden trajectory is governed by two coupled mechanisms: a static tree of self-contained piecewise-linear flow, and a dynamic transport that hops the trajectory across trees when cross-token attention fires. This decomposition yields Geometric Lens, a training-free, hyperparameter-free method for reading out the exact concept a hidden vector encodes at any layer.
We also develop Laguerre Autoencoder, a 2D visualizer that renders both the decision geometry and a model's full reasoning trajectory in one view. Finally, we move beyond explanatory geometry toward actionable interpretability, showing that Geometric Lens recovers the correct factual token when a model is prompted with in-context interference. The code is available on GitHub.
Blogger's Review: Laguerre geometry offers a new mathematical framework for understanding large language models, breaking the limitations of traditional concept representations. This approach not only provides researchers with a clearer view of the internal structures of models but also strengthens the tools for interpretability in practical applications, revealing new possibilities for decoding information in complex models.