Efficient Transformer for Direct Speech Translation
This addresses the computational inefficiency of Transformers for speech translation, though it is incremental as it builds on existing Longformer and Transformer decoder methods.
The paper tackles the problem of long audio sequences in direct speech translation by proposing an efficient Transformer encoder that processes spectrograms directly without convolutional layers, achieving results close to the standard approach.
The advent of Transformer-based models has surpassed the barriers of text. When working with speech, we must face a problem: the sequence length of an audio input is not suitable for the Transformer. To bypass this problem, a usual approach is adding strided convolutional layers, to reduce the sequence length before using the Transformer. In this paper, we propose a new approach for direct Speech Translation, where thanks to an efficient Transformer we can work with a spectrogram without having to use convolutional layers before the Transformer. This allows the encoder to learn directly from the spectrogram and no information is lost. We have created an encoder-decoder model, where the encoder is an efficient Transformer -- the Longformer -- and the decoder is a traditional Transformer decoder. Our results, which are close to the ones obtained with the standard approach, show that this is a promising research direction.