Power-of-Two Quantization-Aware-Training (PoT-QAT) in Large Language Models (LLMs)
This work addresses memory and processing constraints for deploying LLMs on edge devices, presenting an incremental improvement through a specialized quantization method.
The paper tackles the challenge of implementing large language models on edge devices by compressing weights with power-of-two quantization, which reduces memory usage by 87.5% and speeds up inference by 3-10x, while using quantization-aware training to achieve a 66% perplexity improvement and only 1% BERT-Score loss compared to the baseline.
In Large Language Models (LLMs), the number of parameters has grown exponentially in the past few years, e.g., from 1.5 billion parameters in GPT-2 to 175 billion in GPT-3 to possibly more than trillion in higher versions. This raises a significant challenge for implementation, especially for Edge devices. Unlike cloud computing, memory and processing power for Edge devices are very limited, which necessitates developing novel ideas to make such applications feasible. In this work, we investigate compressing weights with a special quantization that limits numbers to only power-of-two (PoT). This helps save a huge amount of memory as only exponents need to be stored, more importantly, it significantly reduces processing power by replacing costly multiplication with low cost bit shifting. To overcome performance loss due to this strict quantization, we investigate Quantization Aware Training (QAT) to enhance performance through additional training. Results on GPT-2 124M show a major enhancement for quantized PoT model after additional training, with a perplexity enhancement of 66% and BERT-Score loss to baseline GPT-2 of 1%. The memory saving is estimated to be 87.5% while the inference speed is expected to be 3-10x faster with PoT quantization versus full-precision.