On Connected Sublevel Sets in Deep Learning
This addresses the optimization landscape problem for deep learning researchers, providing theoretical guarantees for gradient-based methods in over-parameterized networks.
The paper proves that for over-parameterized neural nets with piecewise linear activations, every sublevel set of the loss function is connected and unbounded, implying no bad local valleys and all global minima are connected in a single global valley.
This paper shows that every sublevel set of the loss function of a class of deep over-parameterized neural nets with piecewise linear activation functions is connected and unbounded. This implies that the loss has no bad local valleys and all of its global minima are connected within a unique and potentially very large global valley.