Huff-LLM: End-to-End Lossless Compression for Efficient LLM Inference
This addresses the challenge of deploying state-of-the-art LLMs on resource-constrained edge devices, offering a lossless alternative to existing lossy compression techniques.
The paper tackles the problem of running large language models (LLMs) on edge devices by proposing Huff-LLM, an end-to-end lossless compression method that stores LLM weights in compressed format across all memory levels, resulting in memory savings and improvements in latency and energy efficiency during inference.
As they become more capable, large language models (LLMs) have continued to rapidly increase in size. This has exacerbated the difficulty in running state of the art LLMs on small, edge devices. Standard techniques advocate solving this problem through lossy compression techniques such as quantization or pruning. However, such compression techniques are lossy, and have been shown to change model behavior in unpredictable manners. We propose Huff-LLM, an \emph{end-to-end, lossless} model compression method that lets users store LLM weights in compressed format \emph{everywhere} -- cloud, disk, main memory, and even in on-chip memory/buffers. This allows us to not only load larger models in main memory, but also reduces bandwidth required to load weights on chip, and makes more efficient use of on-chip weight buffers. In addition to the memory savings achieved via compression, we also show latency and energy efficiency improvements when performing inference with the compressed model.