Neural Language Model Pruning for Automatic Speech Recognition
This work addresses efficiency improvements for automatic speech recognition systems, but it is incremental as it builds on existing pruning techniques with specific optimizations.
The paper tackled model pruning for Transformer-based language models in automatic speech recognition, analyzing pruning criteria, methods, and schedulers to improve accuracy and inference speed, with results showing data-driven pruning outperforms magnitude-driven in some cases and incremental pruning achieves higher accuracy for smaller models.
We study model pruning methods applied to Transformer-based neural network language models for automatic speech recognition. We explore three aspects of the pruning frame work, namely criterion, method and scheduler, analyzing their contribution in terms of accuracy and inference speed. To the best of our knowledge, such in-depth analyses on large-scale recognition systems has not been reported in the literature. In addition, we propose a variant of low-rank approximation suitable for incrementally compressing models, and delivering multiple models with varied target sizes. Among other results, we show that a) data-driven pruning outperforms magnitude-driven in several scenarios; b) incremental pruning achieves higher accuracy compared to one-shot pruning, especially when targeting smaller sizes; and c) low-rank approximation presents the best trade-off between size reduction and inference speed-up for moderate compression.