Revisiting Random Weight Perturbation for Efficiently Improving Generalization
This work addresses the challenge of efficient generalization improvement for deep learning practitioners, offering an incremental enhancement to existing RWP methods.
The paper tackles the problem of improving generalization in deep neural networks by revisiting random weight perturbation (RWP), which had lagged behind adversarial methods like SAM; the proposed enhancements achieve comparable or superior performance to SAM with greater efficiency, especially in large-scale problems.
Improving the generalization ability of modern deep neural networks (DNNs) is a fundamental challenge in machine learning. Two branches of methods have been proposed to seek flat minima and improve generalization: one led by sharpness-aware minimization (SAM) minimizes the worst-case neighborhood loss through adversarial weight perturbation (AWP), and the other minimizes the expected Bayes objective with random weight perturbation (RWP). While RWP offers advantages in computation and is closely linked to AWP on a mathematical basis, its empirical performance has consistently lagged behind that of AWP. In this paper, we revisit the use of RWP for improving generalization and propose improvements from two perspectives: i) the trade-off between generalization and convergence and ii) the random perturbation generation. Through extensive experimental evaluations, we demonstrate that our enhanced RWP methods achieve greater efficiency in enhancing generalization, particularly in large-scale problems, while also offering comparable or even superior performance to SAM. The code is released at https://github.com/nblt/mARWP.