On the curvature of the loss landscape
This work addresses a foundational problem in deep learning theory for researchers, but it appears incremental as it builds on existing geometric approaches without demonstrating broad empirical impact.
The paper tackles the challenge of understanding generalization in over-parameterized deep learning models by analyzing the loss landscape as a Riemannian manifold, showing that scalar curvature properties can connect to generalization abilities.
One of the main challenges in modern deep learning is to understand why such over-parameterized models perform so well when trained on finite data. A way to analyze this generalization concept is through the properties of the associated loss landscape. In this work, we consider the loss landscape as an embedded Riemannian manifold and show that the differential geometric properties of the manifold can be used when analyzing the generalization abilities of a deep net. In particular, we focus on the scalar curvature, which can be computed analytically for our manifold, and show connections to several settings that potentially imply generalization.