Masked Audio Text Encoders are Effective Multi-Modal Rescorers
This work addresses domain generalization for ASR systems when target domain data is unavailable, representing an incremental improvement over existing methods.
The paper tackled the problem of domain generalization in Automatic Speech Recognition (ASR) systems by proposing a multi-modal rescorer that incorporates acoustic representations, resulting in word error rate reductions of 4%-16% on in-domain and 3%-7% on out-of-domain datasets.
Masked Language Models (MLMs) have proven to be effective for second-pass rescoring in Automatic Speech Recognition (ASR) systems. In this work, we propose Masked Audio Text Encoder (MATE), a multi-modal masked language model rescorer which incorporates acoustic representations into the input space of MLM. We adopt contrastive learning for effectively aligning the modalities by learning shared representations. We show that using a multi-modal rescorer is beneficial for domain generalization of the ASR system when target domain data is unavailable. MATE reduces word error rate (WER) by 4%-16% on in-domain, and 3%-7% on out-of-domain datasets, over the text-only baseline. Additionally, with very limited amount of training data (0.8 hours), MATE achieves a WER reduction of 8%-23% over the first-pass baseline.