IGC: Integrating a Gated Calculator into an LLM to Solve Arithmetic Tasks Reliably and Efficiently
This addresses a fundamental limitation in LLMs for reliable arithmetic computation, offering a computationally efficient and interpretable solution without external tools.
The paper tackled the problem of LLMs struggling with arithmetic tasks by introducing the Integrated Gated Calculator (IGC), a module that enables LLMs to perform arithmetic efficiently on the GPU, achieving 98-99% accuracy on the BigBench Arithmetic benchmark and outperforming all existing models, including much larger ones.
Solving arithmetic tasks is a simple and fundamental skill, yet modern Large Language Models (LLMs) have great difficulty with them. We introduce the Integrated Gated Calculator (IGC), a module that enables LLMs to perform arithmetic by emulating a calculator on the GPU. We finetune a Llama model with our module and test it on the BigBench Arithmetic benchmark, where it beats the State of the Art, outperforming all models on the benchmark, including models almost two orders of magnitude larger. Our approach takes only a single iteration to run and requires no external tools. It performs arithmetic operations entirely inside the LLM without the need to produce intermediate tokens. It is computationally efficient, interpretable, and avoids side-effects on tasks that do not require arithmetic operations. It reliably achieves 98\% to 99\% accuracy across multiple training runs and for all subtasks, including the substantially harder subtask of multiplication, which was previously unsolved.