Adaptive-saturated RNN: Remember more with less instability
This work addresses a specific problem in sequence learning for researchers and practitioners by offering an incremental improvement to mitigate the vanishing gradient issue in RNNs.
The paper tackles the trade-off between memory capacity and training stability in recurrent neural networks by proposing Adaptive-Saturated RNNs (asRNN), which dynamically adjusts saturation levels to combine the high capacity of vanilla RNNs with the stability of orthogonal RNNs, showing encouraging results on challenging sequence learning benchmarks.
Orthogonal parameterization is a compelling solution to the vanishing gradient problem (VGP) in recurrent neural networks (RNNs). With orthogonal parameters and non-saturated activation functions, gradients in such models are constrained to unit norms. On the other hand, although the traditional vanilla RNNs are seen to have higher memory capacity, they suffer from the VGP and perform badly in many applications. This work proposes Adaptive-Saturated RNNs (asRNN), a variant that dynamically adjusts its saturation level between the two mentioned approaches. Consequently, asRNN enjoys both the capacity of a vanilla RNN and the training stability of orthogonal RNNs. Our experiments show encouraging results of asRNN on challenging sequence learning benchmarks compared to several strong competitors. The research code is accessible at https://github.com/ndminhkhoi46/asRNN/.