Can we learn gradients by Hamiltonian Neural Networks?
This work addresses the challenge of improving optimization flexibility and inductive bias in machine learning tasks, though it appears incremental as it builds on existing meta-learning and neural ODE concepts.
The authors tackled the problem of learning gradients using a meta-learner based on ODE neural networks, specifically Hamiltonian Neural Networks, and demonstrated that it outperforms LSTM-based meta-learners and classic optimization methods on an artificial task and achieves comparable results on MNIST.
In this work, we propose a meta-learner based on ODE neural networks that learns gradients. This approach makes the optimizer is more flexible inducing an automatic inductive bias to the given task. Using the simplest Hamiltonian Neural Network we demonstrate that our method outperforms a meta-learner based on LSTM for an artificial task and the MNIST dataset with ReLU activations in the optimizee. Furthermore, it also surpasses the classic optimization methods for the artificial task and achieves comparable results for MNIST.