The Lattice Representation Hypothesis of Large Language Models
This work addresses a foundational problem in AI by bridging continuous geometry and symbolic abstraction in LLMs, offering insights for interpretability and reasoning, though it is incremental in building on existing hypotheses.
The paper tackles the problem of understanding how large language models (LLMs) encode conceptual hierarchies and logical operations by proposing the Lattice Representation Hypothesis, which unifies linear attribute directions with Formal Concept Analysis to show that LLM embeddings form concept lattices, with experiments on WordNet providing empirical evidence for this geometric-symbolic bridge.
We propose the Lattice Representation Hypothesis of large language models: a symbolic backbone that grounds conceptual hierarchies and logical operations in embedding geometry. Our framework unifies the Linear Representation Hypothesis with Formal Concept Analysis (FCA), showing that linear attribute directions with separating thresholds induce a concept lattice via half-space intersections. This geometry enables symbolic reasoning through geometric meet (intersection) and join (union) operations, and admits a canonical form when attribute directions are linearly independent. Experiments on WordNet sub-hierarchies provide empirical evidence that LLM embeddings encode concept lattices and their logical structure, revealing a principled bridge between continuous geometry and symbolic abstraction.