Hessian-free Optimization for Learning Deep Multidimensional Recurrent Neural Networks
This work addresses the problem of training deep MDRNNs for sequence labeling tasks like speech and handwriting recognition, representing an incremental improvement in optimization methods.
The authors tackled the difficulty of training deep multidimensional recurrent neural networks (MDRNNs) for sequence labeling by using Hessian-free optimization with a convex approximation of connectionist temporal classification (CTC), successfully training networks up to 15 layers deep and achieving improved performance.
Multidimensional recurrent neural networks (MDRNNs) have shown a remarkable performance in the area of speech and handwriting recognition. The performance of an MDRNN is improved by further increasing its depth, and the difficulty of learning the deeper network is overcome by using Hessian-free (HF) optimization. Given that connectionist temporal classification (CTC) is utilized as an objective of learning an MDRNN for sequence labeling, the non-convexity of CTC poses a problem when applying HF to the network. As a solution, a convex approximation of CTC is formulated and its relationship with the EM algorithm and the Fisher information matrix is discussed. An MDRNN up to a depth of 15 layers is successfully trained using HF, resulting in an improved performance for sequence labeling.