Small nonlinearities in activation functions create bad local minima in neural networks
This challenges the robustness of insights from linear networks for practical deep learning, indicating that bad local minima are common in nonlinear settings.
The paper proves that even slight nonlinearities in activation functions cause spurious local minima in neural networks, showing that for ReLU-like networks, almost all datasets have infinitely many local minima, and provides a counterexample for general activations like sigmoid and tanh.
We investigate the loss surface of neural networks. We prove that even for one-hidden-layer networks with "slightest" nonlinearity, the empirical risks have spurious local minima in most cases. Our results thus indicate that in general "no spurious local minima" is a property limited to deep linear networks, and insights obtained from linear networks may not be robust. Specifically, for ReLU(-like) networks we constructively prove that for almost all practical datasets there exist infinitely many local minima. We also present a counterexample for more general activations (sigmoid, tanh, arctan, ReLU, etc.), for which there exists a bad local minimum. Our results make the least restrictive assumptions relative to existing results on spurious local optima in neural networks. We complete our discussion by presenting a comprehensive characterization of global optimality for deep linear networks, which unifies other results on this topic.