Latent Semantic Manifolds in Large Language Models
This work addresses a fundamental problem in understanding LLM geometry for researchers, with implications for architecture design and scaling, but it is incremental as it builds on existing mathematical frameworks.
The paper tackled the geometric mismatch between continuous internal computations and discrete token outputs in large language models by developing a framework that interprets hidden states as points on a latent semantic manifold, and it validated predictions across models, showing universal intrinsic dimension profiles and linear scaling of the expressibility gap with slopes 0.87-1.12.
Large Language Models (LLMs) perform internal computations in continuous vector spaces yet produce discrete tokens -- a fundamental mismatch whose geometric consequences remain poorly understood. We develop a mathematical framework that interprets LLM hidden states as points on a latent semantic manifold: a Riemannian submanifold equipped with the Fisher information metric, where tokens correspond to Voronoi regions partitioning the manifold. We define the expressibility gap, a geometric measure of the semantic distortion from vocabulary discretization, and prove two theorems: a rate-distortion lower bound on distortion for any finite vocabulary, and a linear volume scaling law for the expressibility gap via the coarea formula. We validate these predictions across six transformer architectures (124M-1.5B parameters), confirming universal hourglass intrinsic dimension profiles, smooth curvature structure, and linear gap scaling with slopes 0.87-1.12 (R^2 > 0.985). The margin distribution across models reveals a persistent hard core of boundary-proximal representations invariant to scale, providing a geometric decomposition of perplexity. We discuss implications for architecture design, model compression, decoding strategies, and scaling laws