Sub-8-bit quantization for on-device speech recognition: a regularization-free approach
This work addresses efficiency bottlenecks for on-device automatic speech recognition, offering a novel quantization approach that is incremental over existing methods.
The authors tackled the problem of fixed quantization centroids in quantization-aware training for on-device speech recognition by introducing a regularization-free method with self-adjustable centroids, achieving sub-8-bit compression without accuracy degradation, with results including 30.73% memory saving and 31.75% latency reduction.
For on-device automatic speech recognition (ASR), quantization aware training (QAT) is ubiquitous to achieve the trade-off between model predictive performance and efficiency. Among existing QAT methods, one major drawback is that the quantization centroids have to be predetermined and fixed. To overcome this limitation, we introduce a regularization-free, "soft-to-hard" compression mechanism with self-adjustable centroids in a mu-Law constrained space, resulting in a simpler yet more versatile quantization scheme, called General Quantizer (GQ). We apply GQ to ASR tasks using Recurrent Neural Network Transducer (RNN-T) and Conformer architectures on both LibriSpeech and de-identified far-field datasets. Without accuracy degradation, GQ can compress both RNN-T and Conformer into sub-8-bit, and for some RNN-T layers, to 1-bit for fast and accurate inference. We observe a 30.73% memory footprint saving and 31.75% user-perceived latency reduction compared to 8-bit QAT via physical device benchmarking.