RETURNN as a Generic Flexible Neural Toolkit with Application to Translation and Speech Recognition
This work provides a flexible toolkit for researchers in machine translation and speech recognition, though it appears incremental with optimizations to existing methods.
The authors tackled the challenge of training and decoding speed for attention models in translation and speech recognition, achieving over 1% absolute BLEU improvement with a layer-wise pretraining scheme and state-of-the-art results on WMT 2017 and Switchboard benchmarks.
We compare the fast training and decoding speed of RETURNN of attention models for translation, due to fast CUDA LSTM kernels, and a fast pure TensorFlow beam search decoder. We show that a layer-wise pretraining scheme for recurrent attention models gives over 1% BLEU improvement absolute and it allows to train deeper recurrent encoder networks. Promising preliminary results on max. expected BLEU training are presented. We are able to train state-of-the-art models for translation and end-to-end models for speech recognition and show results on WMT 2017 and Switchboard. The flexibility of RETURNN allows a fast research feedback loop to experiment with alternative architectures, and its generality allows to use it on a wide range of applications.