HeunNet: Extending ResNet using Heun's Methods
This work addresses efficiency and accuracy challenges in deep learning for practitioners, though it is incremental as it builds on existing ResNet and ODE analogies.
The paper tackled the problem of improving deep neural network efficiency by extending ResNet using Heun's method, a predictor-corrector technique from ODE solvers, resulting in HeunNet achieving high accuracy with low computational time compared to vanilla recurrent neural networks and other ResNet variants.
There is an analogy between the ResNet (Residual Network) architecture for deep neural networks and an Euler solver for an ODE. The transformation performed by each layer resembles an Euler step in solving an ODE. We consider the Heun Method, which involves a single predictor-corrector cycle, and complete the analogy, building a predictor-corrector variant of ResNet, which we call a HeunNet. Just as Heun's method is more accurate than Euler's, experiments show that HeunNet achieves high accuracy with low computational (both training and test) time compared to both vanilla recurrent neural networks and other ResNet variants.