Mixed-Precision Training for NLP and Speech Recognition with OpenSeq2Seq
This provides a practical solution for researchers and practitioners in NLP and speech recognition to train models more efficiently, though it is incremental as it builds on existing mixed-precision techniques.
The authors tackled the problem of slow training times for sequence-to-sequence models in NLP and speech recognition by developing OpenSeq2Seq, a toolkit that uses mixed-precision training, resulting in state-of-the-art performance with 1.5-3x faster training.
We present OpenSeq2Seq - a TensorFlow-based toolkit for training sequence-to-sequence models that features distributed and mixed-precision training. Benchmarks on machine translation and speech recognition tasks show that models built using OpenSeq2Seq give state-of-the-art performance at 1.5-3x less training time. OpenSeq2Seq currently provides building blocks for models that solve a wide range of tasks including neural machine translation, automatic speech recognition, and speech synthesis.