Effective and Efficient Mixed Precision Quantization of Speech Foundation Models
This work addresses the need for efficient deployment of large speech models, offering significant compression and speed improvements, though it is incremental as it builds on existing quantization methods.
This paper tackles the problem of compressing speech foundation models by introducing a novel mixed-precision quantization approach that integrates learning and quantization into a single stage, achieving up to 1.9x higher compression ratios and 1.9x faster compression times without increasing word error rates.
This paper presents a novel mixed-precision quantization approach for speech foundation models that tightly integrates mixed-precision learning and quantized model parameter estimation into one single model compression stage. Experiments conducted on LibriSpeech dataset with fine-tuned wav2vec2.0-base and HuBERT-large models suggest the resulting mixed-precision quantized models increased the lossless compression ratio by factors up to 1.7x and 1.9x over the respective uniform-precision and two-stage mixed-precision quantized baselines that perform precision learning and model parameters quantization in separate and disjointed stages, while incurring no statistically word error rate (WER) increase over the 32-bit full-precision models. The system compression time of wav2vec2.0-base and HuBERT-large models is reduced by up to 1.9 and 1.5 times over the two-stage mixed-precision baselines, while both produce lower WERs. The best-performing 3.5-bit mixed-precision quantized HuBERT-large model produces a lossless compression ratio of 8.6x over the 32-bit full-precision system.