A Practical Sparse Approximation for Real Time Recurrent Learning
This work addresses the problem of enabling online weight updates in recurrent neural networks for researchers and practitioners, though it is incremental as it builds on existing RTRL approximations.
The authors tackled the high computational cost of Real Time Recurrent Learning (RTRL) for training recurrent neural networks by introducing the Sparse n-step Approximation (SnAp), which reduces costs to be comparable to backpropagation and outperforms other approximations like Unbiased Online Recurrent Optimization, with SnAp n=2 showing faster learning than backpropagation through time for sparse networks.
Current methods for training recurrent neural networks are based on backpropagation through time, which requires storing a complete history of network states, and prohibits updating the weights `online' (after every timestep). Real Time Recurrent Learning (RTRL) eliminates the need for history storage and allows for online weight updates, but does so at the expense of computational costs that are quartic in the state size. This renders RTRL training intractable for all but the smallest networks, even ones that are made highly sparse. We introduce the Sparse n-step Approximation (SnAp) to the RTRL influence matrix, which only keeps entries that are nonzero within n steps of the recurrent core. SnAp with n=1 is no more expensive than backpropagation, and we find that it substantially outperforms other RTRL approximations with comparable costs such as Unbiased Online Recurrent Optimization. For highly sparse networks, SnAp with n=2 remains tractable and can outperform backpropagation through time in terms of learning speed when updates are done online. SnAp becomes equivalent to RTRL when n is large.