Instance-Based Model Adaptation For Direct Speech Translation
This addresses the data scarcity problem in speech translation for researchers and practitioners, though it is incremental as it builds on existing fine-tuning techniques.
The paper tackles the limited training data for end-to-end speech-to-text translation by proposing an instance-based method to adapt models on-the-fly for each translation request, resulting in coherent performance gains across different languages and data conditions.
Despite recent technology advancements, the effectiveness of neural approaches to end-to-end speech-to-text translation is still limited by the paucity of publicly available training corpora. We tackle this limitation with a method to improve data exploitation and boost the system's performance at inference time. Our approach allows us to customize "on the fly" an existing model to each incoming translation request. At its core, it exploits an instance selection procedure to retrieve, from a given pool of data, a small set of samples similar to the input query in terms of latent properties of its audio signal. The retrieved samples are then used for an instance-specific fine-tuning of the model. We evaluate our approach in three different scenarios. In all data conditions (different languages, in/out-of-domain adaptation), our instance-based adaptation yields coherent performance gains over static models.