Neural Differential Equations for Learning to Program Neural Nets Through Continuous Learning Rules
This work provides a novel method for improving continuous-time neural networks in sequence processing, with potential applications in domains like time series analysis, though it appears incremental as it builds on existing Neural ODE and learning rule frameworks.
The paper tackles the problem of building continuous-time sequence processing networks that learn to manipulate short-term memory in rapidly changing synaptic connections, by combining learning rules with Neural ODEs. The result is that their novel models outperform existing Neural Controlled Differential Equation models on time series classification tasks and address scalability limitations.
Neural ordinary differential equations (ODEs) have attracted much attention as continuous-time counterparts of deep residual neural networks (NNs), and numerous extensions for recurrent NNs have been proposed. Since the 1980s, ODEs have also been used to derive theoretical results for NN learning rules, e.g., the famous connection between Oja's rule and principal component analysis. Such rules are typically expressed as additive iterative update processes which have straightforward ODE counterparts. Here we introduce a novel combination of learning rules and Neural ODEs to build continuous-time sequence processing nets that learn to manipulate short-term memory in rapidly changing synaptic connections of other nets. This yields continuous-time counterparts of Fast Weight Programmers and linear Transformers. Our novel models outperform the best existing Neural Controlled Differential Equation based models on various time series classification tasks, while also addressing their fundamental scalability limitations. Our code is public.