Power-of-Two (PoT) Weights in Large Language Models (LLMs)
This addresses memory and processing constraints for edge devices running LLMs, but is an incremental improvement on existing quantization methods.
The authors tackled the challenge of implementing large language models on edge devices by applying power-of-two quantization to reduce memory and computational complexity, achieving cross-entropy loss degradation between 0.88 and 1.3 with 4-6 bit representations.
Complexity of Neural Networks is increasing rapidly due to the massive increase in model parameters. Specifically, in Large Language Models (LLMs), the number of model parameters has grown exponentially in the past few years, for example, from 1.5 billion parameters in GPT2 to 175 billion in GPT3. This raises a significant challenge for implementation, especially for Edge devices where memory and processing power are very limited. In this work, we investigate reducing LLM complexity with special type of quantization, power of two (PoT), for linear layers weights and transformer tables. PoT not only provides memory reduction but more importantly provides significant computational reduction through converting multiplication to bit shifting. We obtained preliminary results of PoT quantization on Nano-GPT implementation using Shakespeare dataset. We then extended results to 124-M GPT-2 model. The PoT quantization results are shown to be very promising with cross entropy loss degradation $\approx$[1.3-0.88] with number of bits range [4-6] to represent power levels.