LLM Vocabulary Compression for Low-Compute Environments
This enables more efficient deployment of language models in low-compute environments, such as edge devices, but is incremental as it builds on existing compression and BPE techniques.
The paper tackles the problem of high memory usage in language models by compressing the final linear layer, achieving up to 3.4x memory reduction and 3x throughput improvement without significant performance loss, as shown on the TinyStories dataset.
We present a method to compress the final linear layer of language models, reducing memory usage by up to 3.4x without significant performance loss. By grouping tokens based on Byte Pair Encoding (BPE) merges, we prevent materialization of the memory-intensive logits tensor. Evaluations on the TinyStories dataset show that our method performs on par with GPT-Neo and GPT2 while significantly improving throughput by up to 3x, making it suitable for low-compute environments.