Layerwise goal-oriented adaptivity for neural ODEs: an optimal control perspective
This work addresses the challenge of designing efficient neural network architectures for data classification, but it appears incremental as it builds on existing neural ODE and optimal control frameworks.
The authors tackled the problem of constructing neural network architectures adaptively by proposing a layerwise goal-oriented method based on optimal control of neural ODEs, resulting in improved classification performance on well-known datasets, though specific numerical gains are not detailed.
In this work, we propose a novel layerwise adaptive construction method for neural network architectures. Our approach is based on a goal--oriented dual-weighted residual technique for the optimal control of neural differential equations. This leads to an ordinary differential equation constrained optimization problem with controls acting as coefficients and a specific loss function. We implement our approach on the basis of a DG(0) Galerkin discretization of the neural ODE, leading to an explicit Euler time marching scheme. For the optimization we use steepest descent. Finally, we apply our method to the construction of neural networks for the classification of data sets, where we present results for a selection of well known examples from the literature.