Binary Neural Networks for Large Language Model: A Survey
It provides a comprehensive overview for researchers and practitioners in NLP to reduce memory and computational demands in LLMs, but it is incremental as it surveys existing techniques rather than introducing new ones.
This survey paper reviews binary quantization techniques for large language models (LLMs) to address the significant resource overheads from exponential parameter growth, focusing on methods like BitNet that use low-precision binary weights from the start of training.
Large language models (LLMs) have wide applications in the field of natural language processing(NLP), such as GPT-4 and Llama. However, with the exponential growth of model parameter sizes, LLMs bring significant resource overheads. Low-bit quantization, as a key technique, reduces memory usage and computational demands by decreasing the bit-width of model parameters, activations, and gradients. Previous quantization methods for LLMs have largely employed Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT). PTQ does not require any retraining of the original model, while QAT involves optimizing precision during training to achieve the best quantization parameters. The BitNet team proposed a radically different approach, where quantization is performed from the start of model training, utilizing low-precision binary weights during the training process. This approach has led to the emergence of many binary quantization techniques for large language models. This paper provides a comprehensive review of these binary quantization techniques. Specifically, we will introduce binary quantization techniques in deep neural networks and further explore their application to LLMs, reviewing their various contributions, implementations, and applications.