Speech-text based multi-modal training with bidirectional attention for improved speech recognition
This work addresses data efficiency for ASR systems by leveraging unpaired text, which is incremental as it builds on existing multi-modal training approaches.
The paper tackles the problem of improving data efficiency in end-to-end automatic speech recognition (ASR) by enabling multi-modal training with unpaired text data, using a bidirectional attention mechanism to synchronize speech and text features. The result is up to 9.23% word error rate reduction on Librispeech when incorporating unpaired text data.
To let the state-of-the-art end-to-end ASR model enjoy data efficiency, as well as much more unpaired text data by multi-modal training, one needs to address two problems: 1) the synchronicity of feature sampling rates between speech and language (aka text data); 2) the homogeneity of the learned representations from two encoders. In this paper we propose to employ a novel bidirectional attention mechanism (BiAM) to jointly learn both ASR encoder (bottom layers) and text encoder with a multi-modal learning method. The BiAM is to facilitate feature sampling rate exchange, realizing the quality of the transformed features for the one kind to be measured in another space, with diversified objective functions. As a result, the speech representations are enriched with more linguistic information, while the representations generated by the text encoder are more similar to corresponding speech ones, and therefore the shared ASR models are more amenable for unpaired text data pretraining. To validate the efficacy of the proposed method, we perform two categories of experiments with or without extra unpaired text data. Experimental results on Librispeech corpus show it can achieve up to 6.15% word error rate reduction (WERR) with only paired data learning, while 9.23% WERR when more unpaired text data is employed.