Interpolation Technique to Speed Up Gradients Propagation in Neural ODEs
This work addresses a computational bottleneck for researchers and practitioners using neural ODEs, though it appears incremental as it builds on existing gradient approximation techniques.
The authors tackled the problem of slow gradient computation in neural ODEs by proposing an interpolation-based method, which resulted in faster training compared to the reverse dynamic method as validated by numerical experiments on standard benchmarks.
We propose a simple interpolation-based method for the efficient approximation of gradients in neural ODE models. We compare it with the reverse dynamic method (known in the literature as "adjoint method") to train neural ODEs on classification, density estimation, and inference approximation tasks. We also propose a theoretical justification of our approach using logarithmic norm formalism. As a result, our method allows faster model training than the reverse dynamic method that was confirmed and validated by extensive numerical experiments for several standard benchmarks.