ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks
This work addresses a scale-invariance issue in deep learning optimization for improved generalization, but it is incremental as it builds on existing sharpness-aware methods.
The paper tackled the problem of sharpness-based learning algorithms being sensitive to parameter re-scaling, which weakens the connection between sharpness and generalization gap, by introducing adaptive sharpness and ASAM, resulting in significant improvement in model generalization performance across various benchmark datasets.
Recently, learning algorithms motivated from sharpness of loss surface as an effective measure of generalization gap have shown state-of-the-art performances. Nevertheless, sharpness defined in a rigid region with a fixed radius, has a drawback in sensitivity to parameter re-scaling which leaves the loss unaffected, leading to weakening of the connection between sharpness and generalization gap. In this paper, we introduce the concept of adaptive sharpness which is scale-invariant and propose the corresponding generalization bound. We suggest a novel learning method, adaptive sharpness-aware minimization (ASAM), utilizing the proposed generalization bound. Experimental results in various benchmark datasets show that ASAM contributes to significant improvement of model generalization performance.