Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamics
This work addresses a key bottleneck in RNNs for machine learning practitioners by enhancing expressivity without sacrificing stability, though it is incremental relative to existing orthogonal RNN methods.
The authors tackled the trade-off between stability and expressivity in recurrent neural networks (RNNs) by proposing a novel connectivity structure based on Schur decomposition, which allows unit-norm eigenspectra without orthogonality constraints. This resulted in improved expressivity while retaining stability and training speed, as demonstrated on tasks requiring computations over ongoing input sequences.
A recent strategy to circumvent the exploding and vanishing gradient problem in RNNs, and to allow the stable propagation of signals over long time scales, is to constrain recurrent connectivity matrices to be orthogonal or unitary. This ensures eigenvalues with unit norm and thus stable dynamics and training. However this comes at the cost of reduced expressivity due to the limited variety of orthogonal transformations. We propose a novel connectivity structure based on the Schur decomposition and a splitting of the Schur form into normal and non-normal parts. This allows to parametrize matrices with unit-norm eigenspectra without orthogonality constraints on eigenbases. The resulting architecture ensures access to a larger space of spectrally constrained matrices, of which orthogonal matrices are a subset. This crucial difference retains the stability advantages and training speed of orthogonal RNNs while enhancing expressivity, especially on tasks that require computations over ongoing input sequences.