Hierarchical Concept Geometry in Language Models Emerges from Word Co-occurrence
For NLP researchers, this work provides a mechanistic explanation for hierarchical representations in LLMs, showing they arise from simple distributional statistics rather than complex reasoning.
The authors prove that hierarchical concept geometry (hypernymy) in language models emerges from word co-occurrence statistics, not from a dedicated functional mechanism. They confirm this theoretically and empirically in word2vec and Gemma 2B embeddings.
We propose a distributional theory of how hypernymy -- the ``is-a'' relation between general and specific concepts -- is encoded geometrically in language representations. Starting from the empirically verified assumption that words closer on the WordNet hypernym graph co-occur more often, we characterize theoretically the spectrum of the resulting embedding Gram matrix of word2vec embeddings. Under mild positivity and decay conditions on the co-occurrence kernel, we prove that the leading eigenvectors first separate broad taxonomic branches and then progressively finer sub-branches, producing a \emph{hierarchical splitting geometry} with a coarse-to-fine spectral organization that mirrors the tree. We confirm these predictions in word2vec embeddings across many sampled WordNet subtrees, and show that the same signature extends strikingly well to Gemma 2B unembeddings. Our results indicate that hierarchical concept geometry in LLMs need not reflect a hierarchy-specific functional mechanism, but emerges from the spectral structure of pairwise word statistics.