The star-shaped space of solutions of the spherical negative perceptron
This work provides insights into the geometry of solutions in neural networks, which could help understand optimization dynamics, but it is incremental as it builds on prior empirical studies of neural network landscapes.
The authors studied the solution landscape of the spherical negative perceptron, a non-convex neural network model, and found that in the over-parameterized regime, the solution manifold exhibits simple connectivity with a large geodesically convex component and a star-shaped geometry involving high-margin solutions, with a transition at higher constraint densities where this connectivity breaks down.
Empirical studies on the landscape of neural networks have shown that low-energy configurations are often found in complex connected structures, where zero-energy paths between pairs of distant solutions can be constructed. Here we consider the spherical negative perceptron, a prototypical non-convex neural network model framed as a continuous constraint satisfaction problem. We introduce a general analytical method for computing energy barriers in the simplex with vertex configurations sampled from the equilibrium. We find that in the over-parameterized regime the solution manifold displays simple connectivity properties. There exists a large geodesically convex component that is attractive for a wide range of optimization dynamics. Inside this region we identify a subset of atypical high-margin solutions that are geodesically connected with most other solutions, giving rise to a star-shaped geometry. We analytically characterize the organization of the connected space of solutions and show numerical evidence of a transition, at larger constraint densities, where the aforementioned simple geodesic connectivity breaks down.