A Unified Framework of Online Learning Algorithms for Training Recurrent Neural Networks
This work provides a systematic organization for researchers in machine learning and neuroscience to compare online RNN training algorithms, but it is incremental as it synthesizes existing methods without introducing new algorithms.
The authors tackled the problem of organizing and understanding online learning algorithms for training recurrent neural networks by proposing a unified framework that categorizes them based on criteria like past vs. future facing and tensor structure, revealing conceptual connections and showing that performance clusters according to these criteria on synthetic tasks.
We present a framework for compactly summarizing many recent results in efficient and/or biologically plausible online training of recurrent neural networks (RNN). The framework organizes algorithms according to several criteria: (a) past vs. future facing, (b) tensor structure, (c) stochastic vs. deterministic, and (d) closed form vs. numerical. These axes reveal latent conceptual connections among several recent advances in online learning. Furthermore, we provide novel mathematical intuitions for their degree of success. Testing various algorithms on two synthetic tasks shows that performances cluster according to our criteria. Although a similar clustering is also observed for gradient alignment, alignment with exact methods does not alone explain ultimate performance, especially for stochastic algorithms. This suggests the need for better comparison metrics.