Does SGD Seek Flatness or Sharpness? An Exactly Solvable Model
This work clarifies a fundamental debate in deep learning theory about optimization behavior, though it is incremental as it builds on existing hypotheses with a specific model.
The authors tackled the conflicting evidence on whether SGD prefers flat or sharp minima by developing an exactly solvable model, showing that SGD's preference depends on the isotropy of label noise, with isotropic noise leading to flat minima and anisotropic noise to sharp minima.
A large body of theory and empirical work hypothesizes a connection between the flatness of a neural network's loss landscape during training and its performance. However, there have been conceptually opposite pieces of evidence regarding when SGD prefers flatter or sharper solutions during training. In this work, we partially but causally clarify the flatness-seeking behavior of SGD by identifying and exactly solving an analytically solvable model that exhibits both flattening and sharpening behavior during training. In this model, the SGD training has no \textit{a priori} preference for flatness, but only a preference for minimal gradient fluctuations. This leads to the insight that, at least within this model, it is data distribution that uniquely determines the sharpness at convergence, and that a flat minimum is preferred if and only if the noise in the labels is isotropic across all output dimensions. When the noise in the labels is anisotropic, the model instead prefers sharpness and can converge to an arbitrarily sharp solution, depending on the imbalance in the noise in the labels spectrum. We reproduce this key insight in controlled settings with different model architectures such as MLP, RNN, and transformers.