Multiplication-Free Transformer Training via Piecewise Affine Operations
This work addresses the computational efficiency problem for AI practitioners by enabling multiplication-free training of modern architectures, representing an incremental improvement over existing methods.
The paper tackles the high computational cost of multiplications in neural network training by replacing them with piecewise affine approximations using integer bit additions, showing that transformers can be trained on vision and language tasks with minimal performance impact and no hyperparameter changes.
Multiplications are responsible for most of the computational cost involved in neural network training and inference. Recent research has thus looked for ways to reduce the cost associated with them. Inspired by Mogami (2020), we replace multiplication with a cheap piecewise affine approximation that is achieved by adding the bit representation of the floating point numbers together as integers. We show that transformers can be trained with the resulting modified matrix multiplications on both vision and language tasks with little to no performance impact, and without changes to the training hyperparameters. We further replace all non-linearities in the networks making them fully and jointly piecewise affine in both inputs and weights. Finally, we show that we can eliminate all multiplications in the entire training process, including operations in the forward pass, backward pass and optimizer update, demonstrating the first successful training of modern neural network architectures in a fully multiplication-free fashion.