Accelerating recurrent neural network training using sequence bucketing and multi-GPU data parallelization
This work addresses training efficiency for RNNs in tasks with variable-length sequences, such as handwriting recognition, but it is incremental as it builds on existing bucketing and parallelization techniques.
The paper tackles the problem of slow RNN training on variable-length sequences by introducing an algorithm that uses sequence bucketing and multi-GPU data parallelization, resulting in reduced wall clock time and epochs while maintaining validation loss in an online handwriting recognition task.
An efficient algorithm for recurrent neural network training is presented. The approach increases the training speed for tasks where a length of the input sequence may vary significantly. The proposed approach is based on the optimal batch bucketing by input sequence length and data parallelization on multiple graphical processing units. The baseline training performance without sequence bucketing is compared with the proposed solution for a different number of buckets. An example is given for the online handwriting recognition task using an LSTM recurrent neural network. The evaluation is performed in terms of the wall clock time, number of epochs, and validation loss value.