Semi-flat minima and saddle points by embedding neural networks to overparameterization
This work addresses the problem of understanding optimization landscapes in overparameterized neural networks for researchers in machine learning theory, but it is incremental as it builds on existing embedding methods.
The paper theoretically analyzes the training error landscape for overparameterized neural networks by embedding narrower networks into wider ones, finding that smooth and ReLU activations lead to different partially flat landscapes around embedded points, which relates to differences in generalization abilities.
We theoretically study the landscape of the training error for neural networks in overparameterized cases. We consider three basic methods for embedding a network into a wider one with more hidden units, and discuss whether a minimum point of the narrower network gives a minimum or saddle point of the wider one. Our results show that the networks with smooth and ReLU activation have different partially flat landscapes around the embedded point. We also relate these results to a difference of their generalization abilities in overparameterized realization.