StableQuant: Layer Adaptive Post-Training Quantization for Speech Foundation Models
This work addresses the need for efficient deployment of speech foundation models, which is incremental as it adapts existing quantization techniques to a specific domain.
The paper tackles the problem of compressing speech foundation models (SFMs) for automatic speech recognition by proposing StableQuant, a layer-adaptive post-training quantization algorithm that reduces model sizes to a quarter and doubles inference speed while limiting word error rate drop to less than 0.3% with 8-bit quantization.
In this paper, we propose StableQuant, a novel adaptive post-training quantization (PTQ) algorithm for widely used speech foundation models (SFMs). While PTQ has been successfully employed for compressing large language models (LLMs) due to its ability to bypass additional fine-tuning, directly applying these techniques to SFMs may not yield optimal results, as SFMs utilize distinct network architecture for feature extraction. StableQuant demonstrates optimal quantization performance regardless of the network architecture type, as it adaptively determines the quantization range for each layer by analyzing both the scale distributions and overall performance. We evaluate our algorithm on two SFMs, HuBERT and wav2vec2.0, for an automatic speech recognition (ASR) task, and achieve superior performance compared to traditional PTQ methods. StableQuant successfully reduces the sizes of SFM models to a quarter and doubles the inference speed while limiting the word error rate (WER) performance drop to less than 0.3% with 8-bit quantization.