Loss Patterns of Neural Networks
This work provides insights into neural network optimization for researchers, but it is incremental as it builds on existing methods for loss landscape analysis.
The paper tackled the problem of analyzing the loss landscape of neural networks by introducing multi-point optimization to train multiple models simultaneously, and found that the loss surface is diverse and intricate, with batch normalization making it smoother, as demonstrated through experiments on FashionMNIST and CIFAR10 datasets.
We present multi-point optimization: an optimization technique that allows to train several models simultaneously without the need to keep the parameters of each one individually. The proposed method is used for a thorough empirical analysis of the loss landscape of neural networks. By extensive experiments on FashionMNIST and CIFAR10 datasets we demonstrate two things: 1) loss surface is surprisingly diverse and intricate in terms of landscape patterns it contains, and 2) adding batch normalization makes it more smooth. Source code to reproduce all the reported results is available on GitHub: https://github.com/universome/loss-patterns.