Exploring Model Invariance with Discrete Search for Ultra-Low-Bit Quantization
This work addresses the problem of reducing memory usage for large language models to make them more accessible, representing an incremental advance in post-training quantization techniques.
The paper tackles the challenge of ultra-low-bit quantization for large language models by proposing InvarExplore, a framework that explores multiple model invariances simultaneously, including permutation invariance via discrete search, and achieves additional performance improvements over existing state-of-the-art methods.
Large language models have been increasing in size due to their success in a wide range of applications. This calls for a pressing need to reduce memory usage to make them more accessible. Post-training quantization is a popular technique which uses fewer bits (e.g., 4--8 bits) to represent the model without retraining it. However, it remains a challenging task to perform quantization in an ultra-low-bit setup (e.g., 2 bits). In this paper, we propose InvarExplore, a unified framework that systematically explores different model invariance at the same time, allowing us to take advantage of the synergy between each type of invariance. Importantly, InvarExplore features a discrete search algorithm that enables us to explore permutation invariance, which is under-studied as it cannot be optimized with gradient-based methods. Results show that InvarExplore is compatible with existing state-of-the-art methods, achieving an add-on performance improvement over strong competing methods.