Language Models Do Not Embed Numbers Continuously
This work addresses a fundamental issue for users of embedding models in applications requiring high numerical precision, large magnitudes, or mixed-sign values, though it is incremental in building on prior research.
The paper tackled the problem of whether language models embed numbers continuously, finding that they represent numeric spaces as non-continuous with significant noise, as shown by high-fidelity reconstruction (R^2 ≥ 0.95) but low explained variance in principal components.
Recent research has extensively studied how large language models manipulate integers in specific arithmetic tasks, and on a more fundamental level, how they represent numeric values. These previous works have found that language model embeddings can be used to reconstruct the original values, however, they do not evaluate whether language models actually model continuous values as continuous. Using expected properties of the embedding space, including linear reconstruction and principal component analysis, we show that language models not only represent numeric spaces as non-continuous but also introduce significant noise. Using models from three major providers (OpenAI, Google Gemini and Voyage AI), we show that while reconstruction is possible with high fidelity ($R^2 \geq 0.95$), principal components only explain a minor share of variation within the embedding space. This indicates that many components within the embedding space are orthogonal to the simple numeric input space. Further, both linear reconstruction and explained variance suffer with increasing decimal precision, despite the ordinal nature of the input space being fundamentally unchanged. The findings of this work therefore have implications for the many areas where embedding models are used, in-particular where high numerical precision, large magnitudes or mixed-sign values are common.