Sensitivity - Local Index to Control Chaoticity or Gradient Globally -
This work addresses the critical problem of vanishing gradients and chaotic dynamics in deep and recurrent neural networks, which hinders the training of very deep architectures and long-term dependencies, primarily benefiting researchers and practitioners working with such models.
This paper introduces "sensitivity," a local index for each neuron, and a learning method called "sensitivity adjustment learning (SAL)" to control information transmission in neural networks. By adjusting the time average of this index to 1.0 per neuron, the method mitigates chaotic dynamics in RNNs and the vanishing gradient problem in deep feedforward NNs and RNNs, enabling learning in a 300-layer NN or an RNN with a 300-step lag.
Here, we introduce a fully local index named "sensitivity" for each neuron to control chaoticity or gradient globally in a neural network (NN). We also propose a learning method to adjust it named "sensitivity adjustment learning (SAL)". The index is the gradient magnitude of its output with respect to its inputs. By adjusting its time average to 1.0 in each neuron, information transmission in the neuron changes to be moderate without shrinking or expanding for both forward and backward computations. That results in moderate information transmission through a layer of neurons when the weights and inputs are random. Therefore, SAL can control the chaoticity of the network dynamics in a recurrent NN (RNN). It can also solve the vanishing gradient problem in error backpropagation (BP) learning in a deep feedforward NN or an RNN. We demonstrate that when applying SAL to an RNN with small and random initial weights, log-sensitivity, which is the logarithm of RMS (root mean square) sensitivity over all the neurons, is equivalent to the maximum Lyapunov exponent until it reaches 0.0. We also show that SAL works with BP or BPTT (BP through time) to avoid the vanishing gradient problem in a 300-layer NN or an RNN that learns a problem with a lag of 300 steps between the first input and the output. Compared with manually fine-tuning the spectral radius of the weight matrix before learning, SAL's continuous nonlinear learning nature prevents loss of sensitivities during learning, resulting in a significant improvement in learning performance.