Deep Language Geometry: Constructing a Metric Space from LLM Weights
This provides a novel automated approach for linguistic analysis, though it is incremental as it builds on existing LLM and pruning techniques.
The authors tackled the problem of constructing a metric space for languages by using LLM weight activations, resulting in a method that captures linguistic characteristics and reveals connections across 106 languages.
We introduce a novel framework that utilizes the internal weight activations of modern Large Language Models (LLMs) to construct a metric space of languages. Unlike traditional approaches based on hand-crafted linguistic features, our method automatically derives high-dimensional vector representations by computing weight importance scores via an adapted pruning algorithm. Our approach captures intrinsic language characteristics that reflect linguistic phenomena. We validate our approach across diverse datasets and multilingual LLMs, covering 106 languages. The results align well with established linguistic families while also revealing unexpected inter-language connections that may indicate historical contact or language evolution. The source code, computed language latent vectors, and visualization tool are made publicly available at https://github.com/mshamrai/deep-language-geometry.