TokenBlowUp: Resolving Representational Singularities in LLM Token Spaces via Monoidal Transformations
This addresses a foundational issue in LLMs by resolving representational singularities, potentially improving model stability and performance, though it appears incremental as it builds on existing geometric critiques.
The paper tackles the problem of geometric singularities in LLM token embedding spaces, which cause representational instability, by proposing a scheme-theoretic blow-up method that replaces singular points with projective spaces to disambiguate semantics, resulting in a geometrically regularized embedding space as proven by a formal theorem.
Recent work has provided compelling evidence challenging the foundational manifold hypothesis for the token embedding spaces of Large Language Models (LLMs). These findings reveal the presence of geometric singularities around polysemous tokens, which can lead to representational instability. Existing methodologies, which presuppose a smooth data manifold, are ill-equipped to address such intrinsic structural flaws. In this paper, we formalize this problem in the language of scheme theory and propose a rigorous resolution by applying the scheme-theoretic blow-up at each singular point. This procedure replaces a singular point in the ambient affine scheme with its exceptional divisor, which we identify as a canonical geometric space -- a projective space of directions -- that houses the disambiguated semantic meanings of the token. This process of ``representational desingularization'' constructs a new geometric landscape for embeddings. We prove a formal theorem guaranteeing the geometric regularization of this new space, showing that the original pathologies are resolved. Finally, we outline the architectural implications of our framework, arguing for a paradigm shift from static look-ups to dynamic, geometrically-grounded computation.