Globally Optimal Training of Generalized Polynomial Neural Networks with Nonlinear Spectral Methods
This provides a globally optimal training method with guarantees for neural networks, addressing a fundamental challenge in machine learning, though it is currently limited to shallow networks.
The authors tackled the non-convex optimization problem in neural network training by proposing a nonlinear spectral method that achieves global optimality with linear convergence for a class of feedforward networks, as confirmed by experiments on real-world datasets.
The optimization problem behind neural networks is highly non-convex. Training with stochastic gradient descent and variants requires careful parameter tuning and provides no guarantee to achieve the global optimum. In contrast we show under quite weak assumptions on the data that a particular class of feedforward neural networks can be trained globally optimal with a linear convergence rate with our nonlinear spectral method. Up to our knowledge this is the first practically feasible method which achieves such a guarantee. While the method can in principle be applied to deep networks, we restrict ourselves for simplicity in this paper to one and two hidden layer networks. Our experiments confirm that these models are rich enough to achieve good performance on a series of real-world datasets.