Fast Saturating Gate for Learning Long Time Scales with Recurrent Neural Networks
This work addresses a bottleneck in training recurrent models for time series with long dependencies, offering an incremental improvement over existing gate functions like those in LSTMs and GRUs.
The authors tackled the saturation problem in recurrent neural network gates, which hinders learning of long time scales due to gradient vanishing, by proposing a fast gate function with doubly exponential convergence. They empirically demonstrated improved accuracy and computational efficiency on benchmark tasks involving extremely long time scales.
Gate functions in recurrent models, such as an LSTM and GRU, play a central role in learning various time scales in modeling time series data by using a bounded activation function. However, it is difficult to train gates to capture extremely long time scales due to gradient vanishing of the bounded function for large inputs, which is known as the saturation problem. We closely analyze the relation between saturation of the gate function and efficiency of the training. We prove that the gradient vanishing of the gate function can be mitigated by accelerating the convergence of the saturating function, i.e., making the output of the function converge to 0 or 1 faster. Based on the analysis results, we propose a gate function called fast gate that has a doubly exponential convergence rate with respect to inputs by simple function composition. We empirically show that our method outperforms previous methods in accuracy and computational efficiency on benchmark tasks involving extremely long time scales.