Using External Off-Policy Speech-To-Text Mappings in Contextual End-To-End Automated Speech Recognition
This work addresses the challenge of adapting production ASR systems to new data distributions with reduced data availability and shifting distributions, offering a solution for zero and few-shot scenarios.
The paper tackled the problem of specializing automated speech recognition (ASR) models for downstream tasks by leveraging external off-policy key-value stores from text-to-speech to enable flexible post-training adaptation, resulting in up to 3% WER improvement and a reduction of domain adaptation time by up to 1K GPU-hours compared to fine-tuning baselines.
Despite improvements to the generalization performance of automated speech recognition (ASR) models, specializing ASR models for downstream tasks remains a challenging task, primarily due to reduced data availability (necessitating increased data collection), and rapidly shifting data distributions (requiring more frequent model fine-tuning). In this work, we investigate the potential of leveraging external knowledge, particularly through off-policy key-value stores generated with text-to-speech methods, to allow for flexible post-training adaptation to new data distributions. In our approach, audio embeddings captured from text-to-speech, along with semantic text embeddings, are used to bias ASR via an approximate k-nearest-neighbor (KNN) based attentive fusion step. Our experiments on LibiriSpeech and in-house voice assistant/search datasets show that the proposed approach can reduce domain adaptation time by up to 1K GPU-hours while providing up to 3% WER improvement compared to a fine-tuning baseline, suggesting a promising approach for adapting production ASR systems in challenging zero and few-shot scenarios.