AdaTranS: Adapting with Boundary-based Shrinking for End-to-End Speech Translation
This work addresses data scarcity in end-to-end speech translation, an incremental improvement for researchers and practitioners in speech processing.
The paper tackles the modality gap and length mismatch between speech and text in end-to-end speech translation by proposing AdaTranS, which adapts speech features using a boundary-based shrinking mechanism, achieving better performance than other shrinking-based methods on the MUST-C dataset with higher inference speed and lower memory usage.
To alleviate the data scarcity problem in End-to-end speech translation (ST), pre-training on data for speech recognition and machine translation is considered as an important technique. However, the modality gap between speech and text prevents the ST model from efficiently inheriting knowledge from the pre-trained models. In this work, we propose AdaTranS for end-to-end ST. It adapts the speech features with a new shrinking mechanism to mitigate the length mismatch between speech and text features by predicting word boundaries. Experiments on the MUST-C dataset demonstrate that AdaTranS achieves better performance than the other shrinking-based methods, with higher inference speed and lower memory usage. Further experiments also show that AdaTranS can be equipped with additional alignment losses to further improve performance.