Continual Learning for On-Device Speech Recognition using Disentangled Conformers
This work addresses the challenge of speaker-specific adaptation for on-device speech recognition, offering a novel benchmark and method, but it is incremental as it builds on existing continual learning and model adaptation techniques.
The paper tackles the problem of adapting speech recognition models to user-specific distributional shifts on personal devices by introducing a continual learning benchmark, LibriContinual, and a compute-efficient algorithm, DisentangledCL, which uses DisConformer models to achieve significant improvements, such as a 20.65% relative WER reduction on speaker-specific tasks.
Automatic speech recognition research focuses on training and evaluating on static datasets. Yet, as speech models are increasingly deployed on personal devices, such models encounter user-specific distributional shifts. To simulate this real-world scenario, we introduce LibriContinual, a continual learning benchmark for speaker-specific domain adaptation derived from LibriVox audiobooks, with data corresponding to 118 individual speakers and 6 train splits per speaker of different sizes. Additionally, current speech recognition models and continual learning algorithms are not optimized to be compute-efficient. We adapt a general-purpose training algorithm NetAug for ASR and create a novel Conformer variant called the DisConformer (Disentangled Conformer). This algorithm produces ASR models consisting of a frozen 'core' network for general-purpose use and several tunable 'augment' networks for speaker-specific tuning. Using such models, we propose a novel compute-efficient continual learning algorithm called DisentangledCL. Our experiments show that the DisConformer models significantly outperform baselines on general ASR i.e. LibriSpeech (15.58% rel. WER on test-other). On speaker-specific LibriContinual they significantly outperform trainable-parameter-matched baselines (by 20.65% rel. WER on test) and even match fully finetuned baselines in some settings.