LyaNet: A Lyapunov Framework for Training Neural ODEs
This work addresses stability and efficiency issues in neural ODE training, which is important for researchers and practitioners in machine learning, though it appears incremental as it builds on existing Neural ODE methods with a control-theoretic twist.
The authors tackled the problem of training neural ordinary differential equations (ODEs) by introducing LyaNet, a method based on a Lyapunov framework that ensures stability and faster convergence, resulting in improved prediction performance, faster inference dynamics, and enhanced adversarial robustness compared to standard training.
We propose a method for training ordinary differential equations by using a control-theoretic Lyapunov condition for stability. Our approach, called LyaNet, is based on a novel Lyapunov loss formulation that encourages the inference dynamics to converge quickly to the correct prediction. Theoretically, we show that minimizing Lyapunov loss guarantees exponential convergence to the correct solution and enables a novel robustness guarantee. We also provide practical algorithms, including one that avoids the cost of backpropagating through a solver or using the adjoint method. Relative to standard Neural ODE training, we empirically find that LyaNet can offer improved prediction performance, faster convergence of inference dynamics, and improved adversarial robustness. Our code available at https://github.com/ivandariojr/LyapunovLearning .