Carrying over algorithm in transformers
This work addresses the interpretability of transformer models in performing basic arithmetic, offering insights into their internal mechanisms, though it is incremental as it builds on existing understanding of modularity in neural networks.
The paper investigates how transformer models implement the carrying over algorithm for addition, showing that it is modularly allocated across network layers, with the first layer handling digit addition and the second layer managing carry decisions and execution, and provides evidence of this implementation across various model sizes, including large language models.
Addition is perhaps one of the simplest arithmetic tasks one can think of and is usually performed using the carrying over algorithm. This algorithm consists of two tasks: adding digits in the same position and carrying over a one whenever necessary. We study how transformer models implement this algorithm and how the two aforementioned tasks are allocated to different parts of the network. We first focus on two-layer encoder-only models and show that the carrying over algorithm is implemented in a modular fashion. The first layer is mostly responsible for adding digits in the same position. The second layer first decides, in the attention, which positions need a carried one or not, and then performs the carrying of the one in the final MLP. We provide a simple way of precisely identifying which neurons are responsible for that task. This implementation of the carrying over algorithm occurs across a range of hyperparameters for two as well as three-layer models. For small decoder-only models, we observe the same implementation and provide suggestive evidence for its existence in three 7B large language models.