Minimum sharpness: Scale-invariant parameter-robustness of neural networks
This work addresses a critical problem in robust neural network design for researchers and practitioners, though it is incremental as it builds on prior sharpness concepts.
The paper tackles the scale-sensitivity issue in neural network sharpness measures by proposing Minimum Sharpness, a scale-invariant measure that correlates with generalization and reduces computational costs compared to existing methods.
Toward achieving robust and defensive neural networks, the robustness against the weight parameters perturbations, i.e., sharpness, attracts attention in recent years (Sun et al., 2020). However, sharpness is known to remain a critical issue, "scale-sensitivity." In this paper, we propose a novel sharpness measure, Minimum Sharpness. It is known that NNs have a specific scale transformation that constitutes equivalent classes where functional properties are completely identical, and at the same time, their sharpness could change unlimitedly. We define our sharpness through a minimization problem over the equivalent NNs being invariant to the scale transformation. We also develop an efficient and exact technique to make the sharpness tractable, which reduces the heavy computational costs involved with Hessian. In the experiment, we observed that our sharpness has a valid correlation with the generalization of NNs and runs with less computational cost than existing sharpness measures.