RAND: Robustness Aware Norm Decay For Quantized Seq2seq Models
This work addresses model compression for sequence tasks, which is incremental as it builds on existing quantization-aware training methods but focuses on under-explored domains like speech recognition.
The paper tackled the problem of quantization for sequence-to-sequence models, particularly in speech recognition, by benchmarking existing techniques and proposing low-complexity changes to quantization-aware training, resulting in improved accuracy that outperforms popular methods and enables benefits like mixed-precision performance and better generalization on long-form speech.
With the rapid increase in the size of neural networks, model compression has become an important area of research. Quantization is an effective technique at decreasing the model size, memory access, and compute load of large models. Despite recent advances in quantization aware training (QAT) technique, most papers present evaluations that are focused on computer vision tasks, which have different training dynamics compared to sequence tasks. In this paper, we first benchmark the impact of popular techniques such as straight through estimator, pseudo-quantization noise, learnable scale parameter, clipping, etc. on 4-bit seq2seq models across a suite of speech recognition datasets ranging from 1,000 hours to 1 million hours, as well as one machine translation dataset to illustrate its applicability outside of speech. Through the experiments, we report that noise based QAT suffers when there is insufficient regularization signal flowing back to the quantization scale. We propose low complexity changes to the QAT process to improve model accuracy (outperforming popular learnable scale and clipping methods). With the improved accuracy, it opens up the possibility to exploit some of the other benefits of noise based QAT: 1) training a single model that performs well in mixed precision mode and 2) improved generalization on long form speech recognition.