Unravelling the Mechanisms of Manipulating Numbers in Language Models
This provides a fundamental understanding of how pre-trained LLMs handle numbers, which is incremental for improving model architectures and probing techniques.
The paper tackled the conflict between language models having accurate number representations but producing errors, finding that models learn systematic and interchangeable number representations across hidden states and contexts, enabling universal probes to trace error causes to specific layers.
Recent work has shown that different large language models (LLMs) converge to similar and accurate input embedding representations for numbers. These findings conflict with the documented propensity of LLMs to produce erroneous outputs when dealing with numeric information. In this work, we aim to explain this conflict by exploring how language models manipulate numbers and quantify the lower bounds of accuracy of these mechanisms. We find that despite surfacing errors, different language models learn interchangeable representations of numbers that are systematic, highly accurate and universal across their hidden states and the types of input contexts. This allows us to create universal probes for each LLM and to trace information -- including the causes of output errors -- to specific layers. Our results lay a fundamental understanding of how pre-trained LLMs manipulate numbers and outline the potential of more accurate probing techniques in addressed refinements of LLMs' architectures.