Learning by solving differential equations
This work addresses the problem of inefficient optimization in deep learning for researchers and practitioners by offering incremental improvements to existing gradient-based methods.
The paper tackled the limited use of higher-order Runge-Kutta ODE solvers in deep learning by evaluating their performance, studying limitations, and proposing improvements through integration of modern optimizer features like preconditioning and adaptive learning rates, resulting in enhanced stability and efficiency in training neural networks.
Modern deep learning algorithms use variations of gradient descent as their main learning methods. Gradient descent can be understood as the simplest Ordinary Differential Equation (ODE) solver; namely, the Euler method applied to the gradient flow differential equation. Since Euler, many ODE solvers have been devised that follow the gradient flow equation more precisely and more stably. Runge-Kutta (RK) methods provide a family of very powerful explicit and implicit high-order ODE solvers. However, these higher-order solvers have not found wide application in deep learning so far. In this work, we evaluate the performance of higher-order RK solvers when applied in deep learning, study their limitations, and propose ways to overcome these drawbacks. In particular, we explore how to improve their performance by naturally incorporating key ingredients of modern neural network optimizers such as preconditioning, adaptive learning rates, and momentum.