Quantized Evolution Strategies: High-precision Fine-tuning of Quantized LLMs at Low-precision Cost
This addresses the challenge of adapting quantized LLMs for deployment on memory-constrained devices, enabling fine-tuning without high-precision weights, which is incremental as it builds on evolution strategies with novel enhancements.
The paper tackles the problem of fine-tuning quantized large language models (LLMs), which are non-differentiable and static after post-training quantization, by introducing Quantized Evolution Strategies (QES) that enables full-parameter fine-tuning directly in quantized space, resulting in significant outperformance over state-of-the-art zeroth-order methods on arithmetic reasoning tasks.
Post-Training Quantization (PTQ) is essential for deploying Large Language Models (LLMs) on memory-constrained devices, yet it renders models static and difficult to fine-tune. Standard fine-tuning paradigms, including Reinforcement Learning (RL), fundamentally rely on backpropagation and high-precision weights to compute gradients. Thus they cannot be used on quantized models, where the parameter space is discrete and non-differentiable. While Evolution Strategies (ES) offer a backpropagation-free alternative, optimization of the quantized parameters can still fail due to vanishing or inaccurate gradient. This paper introduces Quantized Evolution Strategies (QES), an optimization paradigm that performs full-parameter fine-tuning directly in the quantized space. QES is based on two innovations: (1) it integrates accumulated error feedback to preserve high-precision gradient signals, and (2) it utilizes a stateless seed replay to reduce memory usage to low-precision inference levels. QES significantly outperforms the state-of-the-art zeroth-order fine-tuning method on arithmetic reasoning tasks, making direct fine-tuning for quantized models possible. It therefore opens up the possibility for scaling up LLMs entirely in the quantized space. The source code is available at https://github.com/dibbla/Quantized-Evolution-Strategies .