On the overfly algorithm in deep learning of neural networks
This addresses the local minima issue in neural network training, which is a fundamental challenge for improving optimization in deep learning.
The paper tackles the local minima problem in supervised backpropagation training of multilayer neural networks by introducing the overfly algorithm, which leverages dynamical systems theory and first integrals of generalized gradient systems with dissipation.
In this paper we investigate the supervised backpropagation training of multilayer neural networks from a dynamical systems point of view. We discuss some links with the qualitative theory of differential equations and introduce the overfly algorithm to tackle the local minima problem. Our approach is based on the existence of first integrals of the generalised gradient system with build-in dissipation.