Backpropagation Through Time For Networks With Long-Term Dependencies
This addresses a potential bottleneck for advancing RNNs, though it is incremental as it builds on existing backpropagation techniques.
The paper tackles the problem of training recurrent neural networks (RNNs) with long-term dependencies, proposing an exact method using discrete forward sensitivity equations that allows parameter variation between steps but requires Jacobian computation.
Backpropagation through time (BPTT) is a technique of updating tuned parameters within recurrent neural networks (RNNs). Several attempts at creating such an algorithm have been made including: Nth Ordered Approximations and Truncated-BPTT. These methods approximate the backpropagation gradients under the assumption that the RNN only utilises short-term dependencies. This is an acceptable assumption to make for the current state of artificial neural networks. As RNNs become more advanced, a shift towards influence by long-term dependencies is likely. Thus, a new method for backpropagation is required. We propose using the 'discrete forward sensitivity equation' and a variant of it for single and multiple interacting recurrent loops respectively. This solution is exact and also allows the network's parameters to vary between each subsequent step, however it does require the computation of a Jacobian.