On Linear Stability of SGD and Input-Smoothness of Neural Networks
This work provides theoretical insights into why flat minima generalize well and how SGD implicitly regularizes neural networks, which is foundational for understanding optimization and generalization in deep learning.
The paper connects the flatness of loss minima to the smoothness of neural network outputs, showing that flat minima regularize model gradients and thus improve generalization. It further demonstrates that SGD imposes Sobolev regularization on the model function, leading to bounds on generalization error and adversarial robustness under certain data assumptions.
The multiplicative structure of parameters and input data in the first layer of neural networks is explored to build connection between the landscape of the loss function with respect to parameters and the landscape of the model function with respect to input data. By this connection, it is shown that flat minima regularize the gradient of the model function, which explains the good generalization performance of flat minima. Then, we go beyond the flatness and consider high-order moments of the gradient noise, and show that Stochastic Gradient Descent (SGD) tends to impose constraints on these moments by a linear stability analysis of SGD around global minima. Together with the multiplicative structure, we identify the Sobolev regularization effect of SGD, i.e. SGD regularizes the Sobolev seminorms of the model function with respect to the input data. Finally, bounds for generalization error and adversarial robustness are provided for solutions found by SGD under assumptions of the data distribution.