Qualitatively characterizing neural network optimization problems
This addresses the long-standing belief in the difficulty of non-convex optimization for neural networks, showing it may be less problematic than assumed, which is incremental but clarifies a key issue for machine learning practitioners.
The paper tackles the problem of neural network optimization by investigating whether training encounters significant local minima, finding that modern networks achieve negligible training error without encountering obstacles on a straight path from initialization to solution.
Training neural networks involves solving large-scale non-convex optimization problems. This task has long been believed to be extremely difficult, with fear of local minima and other obstacles motivating a variety of schemes to improve optimization, such as unsupervised pretraining. However, modern neural networks are able to achieve negligible training error on complex tasks, using only direct training with stochastic gradient descent. We introduce a simple analysis technique to look for evidence that such networks are overcoming local optima. We find that, in fact, on a straight path from initialization to solution, a variety of state of the art neural networks never encounter any significant obstacles.