Inter-connection: Effective Connection between Pre-trained Encoder and Decoder for Speech Translation
This work addresses a bottleneck in speech translation for improving translation quality by better leveraging pre-trained models, though it is incremental as it builds on existing encoder-decoder frameworks.
The paper tackles the problem of underutilizing layer-wise information in speech pre-trained models for speech translation by proposing an inter-connection mechanism that aggregates outputs from each layer, resulting in a BLEU score increase of approximately 2 points for en-de, en-ja, and en-zh with only 2K additional parameters.
In end-to-end speech translation, speech and text pre-trained models improve translation quality. Recently proposed models simply connect the pre-trained models of speech and text as encoder and decoder. Therefore, only the information from the final layer of encoders is input to the decoder. Since it is clear that the speech pre-trained model outputs different information from each layer, the simple connection method cannot fully utilize the information that the speech pre-trained model has. In this study, we propose an inter-connection mechanism that aggregates the information from each layer of the speech pre-trained model by weighted sums and inputs into the decoder. This mechanism increased BLEU by approximately 2 points in en-de, en-ja, and en-zh by increasing parameters by 2K when the speech pre-trained model was frozen. Furthermore, we investigated the contribution of each layer for each language by visualizing layer weights and found that the contributions were different.