Number Representations in LLMs: A Computational Parallel to Human Perception
This provides insights into how LLMs process numerical information, which could inform model interpretability and design for tasks involving numbers.
The paper investigated whether large language models (LLMs) encode numbers in a logarithmic-like structure similar to human perception, and found that their numerical representations exhibit sublinear spacing aligning with a logarithmic scale.
Humans are believed to perceive numbers on a logarithmic mental number line, where smaller values are represented with greater resolution than larger ones. This cognitive bias, supported by neuroscience and behavioral studies, suggests that numerical magnitudes are processed in a sublinear fashion rather than on a uniform linear scale. Inspired by this hypothesis, we investigate whether large language models (LLMs) exhibit a similar logarithmic-like structure in their internal numerical representations. By analyzing how numerical values are encoded across different layers of LLMs, we apply dimensionality reduction techniques such as PCA and PLS followed by geometric regression to uncover latent structures in the learned embeddings. Our findings reveal that the model's numerical representations exhibit sublinear spacing, with distances between values aligning with a logarithmic scale. This suggests that LLMs, much like humans, may encode numbers in a compressed, non-uniform manner.