Evaluating Transformer Language Models on Arithmetic Operations Using Number Decomposition
This addresses the reasoning limitations in language models for arithmetic tasks, though it is incremental as it builds on existing fine-tuning methods.
The paper tackled the problem of Transformer Language Models struggling with arithmetic operations by introducing a pipeline that decomposes numbers before computation, resulting in a 63% accuracy increase in five-digit addition compared to GPT-3.
In recent years, Large Language Models such as GPT-3 showed remarkable capabilities in performing NLP tasks in the zero and few shot settings. On the other hand, the experiments highlighted the difficulty of GPT-3 in carrying out tasks that require a certain degree of reasoning, such as arithmetic operations. In this paper we evaluate the ability of Transformer Language Models to perform arithmetic operations following a pipeline that, before performing computations, decomposes numbers in units, tens, and so on. We denote the models fine-tuned with this pipeline with the name Calculon and we test them in the task of performing additions, subtractions and multiplications on the same test sets of GPT-3. Results show an increase of accuracy of 63% in the five-digit addition task. Moreover, we demonstrate the importance of the decomposition pipeline introduced, since fine-tuning the same Language Model without decomposing numbers results in 0% accuracy in the five-digit addition task.