Revisiting "Qualitatively Characterizing Neural Network Optimization Problems"
This work is significant for researchers and practitioners in deep learning, as it challenges a previously held understanding of the optimization landscape for modern neural networks, suggesting that the "simple, approximately convex shape" observed in earlier work does not generalize to current, more complex settings.
This paper re-examines the findings of Goodfellow et al. (2014) regarding the convexity of neural network optimization paths. Contrary to the original study, this research found that for modern networks on CIFAR-10 and ImageNet, the loss along the linear path between initialization and trained weights remains high until very close to the optimum, and training paths quickly become separated from the optimum by loss barriers.
We revisit and extend the experiments of Goodfellow et al. (2014), who showed that - for then state-of-the-art networks - "the objective function has a simple, approximately convex shape" along the linear path between initialization and the trained weights. We do not find this to be the case for modern networks on CIFAR-10 and ImageNet. Instead, although loss is roughly monotonically non-increasing along this path, it remains high until close to the optimum. In addition, training quickly becomes linearly separated from the optimum by loss barriers. We conclude that, although Goodfellow et al.'s findings describe the "relatively easy to optimize" MNIST setting, behavior is qualitatively different in modern settings.