On the Compressibility of Quantized Large Language Models
This work addresses the problem of efficient LLM deployment for edge computing, but it is incremental as it builds on existing quantization and compression methods.
The paper tackles the challenge of deploying quantized large language models (LLMs) on memory-constrained edge or mobile devices, where I/O latency from model loading becomes a bottleneck, by exploring data compression techniques to reduce data movement and speed up inference.
Deploying Large Language Models (LLMs) on edge or mobile devices offers significant benefits, such as enhanced data privacy and real-time processing capabilities. However, it also faces critical challenges due to the substantial memory requirement of LLMs. Quantization is an effective way of reducing the model size while maintaining good performance. However, even after quantization, LLMs may still be too big to fit entirely into the limited memory of edge or mobile devices and have to be partially loaded from the storage to complete the inference. In this case, the I/O latency of model loading becomes the bottleneck of the LLM inference latency. In this work, we take a preliminary step of studying applying data compression techniques to reduce data movement and thus speed up the inference of quantized LLM on memory-constrained devices. In particular, we discussed the compressibility of quantized LLMs, the trade-off between the compressibility and performance of quantized LLMs, and opportunities to optimize both of them jointly.