fairseq S2T: Fast Speech-to-Text Modeling with fairseq
This work offers a scalable and extensible toolkit for researchers and practitioners in speech processing, but it is incremental as it builds on existing fairseq frameworks.
The authors introduced fairseq S2T, an extension for speech-to-text modeling tasks like speech recognition and translation, providing end-to-end workflows and implementing state-of-the-art models with open-source training recipes.
We introduce fairseq S2T, a fairseq extension for speech-to-text (S2T) modeling tasks such as end-to-end speech recognition and speech-to-text translation. It follows fairseq's careful design for scalability and extensibility. We provide end-to-end workflows from data pre-processing, model training to offline (online) inference. We implement state-of-the-art RNN-based, Transformer-based as well as Conformer-based models and open-source detailed training recipes. Fairseq's machine translation models and language models can be seamlessly integrated into S2T workflows for multi-task learning or transfer learning. Fairseq S2T documentation and examples are available at https://github.com/pytorch/fairseq/tree/master/examples/speech_to_text.