Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscape
This addresses the challenge of poor generalization in deep learning models by developing a method to recover flat optima, representing an incremental improvement over existing optimization techniques.
The paper tackles the problem of finding flat optima in deep learning loss landscapes to improve generalization, and introduces LPF-SGD, an algorithm that achieves superior generalization performance compared to common training strategies, with theoretical convergence guarantees.
In this paper, we study the sharpness of a deep learning (DL) loss landscape around local minima in order to reveal systematic mechanisms underlying the generalization abilities of DL models. Our analysis is performed across varying network and optimizer hyper-parameters, and involves a rich family of different sharpness measures. We compare these measures and show that the low-pass filter-based measure exhibits the highest correlation with the generalization abilities of DL models, has high robustness to both data and label noise, and furthermore can track the double descent behavior for neural networks. We next derive the optimization algorithm, relying on the low-pass filter (LPF), that actively searches the flat regions in the DL optimization landscape using SGD-like procedure. The update of the proposed algorithm, that we call LPF-SGD, is determined by the gradient of the convolution of the filter kernel with the loss function and can be efficiently computed using MC sampling. We empirically show that our algorithm achieves superior generalization performance compared to the common DL training strategies. On the theoretical front, we prove that LPF-SGD converges to a better optimal point with smaller generalization error than SGD.