Exploring Neural Network Landscapes: Star-Shaped and Geodesic Connectivity
This work provides theoretical insights into neural network optimization for researchers, though it is incremental in building on existing mode connectivity concepts.
The paper tackled the problem of understanding connectivity between minima in neural network landscapes, demonstrating that overparameterized networks can have simple linear connecting paths with lengths nearly equal to Euclidean distances and revealing star-shaped connectivity where multiple minima connect via a central point.
One of the most intriguing findings in the structure of neural network landscape is the phenomenon of mode connectivity: For two typical global minima, there exists a path connecting them without barrier. This concept of mode connectivity has played a crucial role in understanding important phenomena in deep learning. In this paper, we conduct a fine-grained analysis of this connectivity phenomenon. First, we demonstrate that in the overparameterized case, the connecting path can be as simple as a two-piece linear path, and the path length can be nearly equal to the Euclidean distance. This finding suggests that the landscape should be nearly convex in a certain sense. Second, we uncover a surprising star-shaped connectivity: For a finite number of typical minima, there exists a center on minima manifold that connects all of them simultaneously via linear paths. These results are provably valid for linear networks and two-layer ReLU networks under a teacher-student setup, and are empirically supported by models trained on MNIST and CIFAR-10.