Training Input-Output Recurrent Neural Networks through Spectral Methods
This provides a theoretically-grounded training approach for RNNs in sequence labeling tasks like POS tagging, though it appears incremental as spectral methods are known in other contexts.
The authors tackled the problem of training input-output recurrent neural networks for sequence labeling by proposing a spectral method based on cross-moment tensor decomposition, guaranteeing consistent learning with polynomial sample and computational complexity under conditions like non-degenerate parameters and polynomial activations.
We consider the problem of training input-output recurrent neural networks (RNN) for sequence labeling tasks. We propose a novel spectral approach for learning the network parameters. It is based on decomposition of the cross-moment tensor between the output and a non-linear transformation of the input, based on score functions. We guarantee consistent learning with polynomial sample and computational complexity under transparent conditions such as non-degeneracy of model parameters, polynomial activations for the neurons, and a Markovian evolution of the input sequence. We also extend our results to Bidirectional RNN which uses both previous and future information to output the label at each time point, and is employed in many NLP tasks such as POS tagging.