When are 1.58 bits enough? A Bottom-up Exploration of BitNet Quantization
This work addresses the resource efficiency problem for machine learning practitioners by showing that 1.58-bit quantization is effective across multiple model types, though it is incremental as it extends prior findings on ternary weights.
The paper tackled the problem of high resource requirements in machine learning models by exploring 1.58-bit quantization training across various architectures, including multi-layer perceptrons, graph neural networks, and transformer-based language models, and found it performs on par with or better than standard 32/16-bit models.
Contemporary machine learning models, such as language models, are powerful, but come with immense resource requirements both at training and inference time. It has been shown that decoder-only language models can be trained to a competitive state with ternary weights (1.58 bits per weight), facilitating efficient inference. Here, we start our exploration with non-transformer model architectures, investigating 1.58-bit training for multi-layer perceptrons and graph neural networks. Then, we explore 1.58-bit training in other transformer-based language models, namely encoder-only and encoder-decoder models. Our results show that in all of these settings, 1.58-bit training is on par with or sometimes even better than the standard 32/16-bit models.