Stationary Point Losses for Robust Model
This addresses the problem of model vulnerability for security-demanding applications, offering a novel loss function to enhance robustness.
The paper tackles the lack of robustness guarantees in deep learning models by identifying that cross-entropy loss fails to ensure robust decision boundaries, and proposes stationary point (SP) loss to guarantee robust boundaries without significant accuracy loss, resulting in improved robustness against adversarial attacks and better performance on imbalanced datasets.
The inability to guarantee robustness is one of the major obstacles to the application of deep learning models in security-demanding domains. We identify that the most commonly used cross-entropy (CE) loss does not guarantee robust boundary for neural networks. CE loss sharpens the neural network at the decision boundary to achieve a lower loss, rather than pushing the boundary to a more robust position. A robust boundary should be kept in the middle of samples from different classes, thus maximizing the margins from the boundary to the samples. We think this is due to the fact that CE loss has no stationary point. In this paper, we propose a family of new losses, called stationary point (SP) loss, which has at least one stationary point on the correct classification side. We proved that robust boundary can be guaranteed by SP loss without losing much accuracy. With SP loss, larger perturbations are required to generate adversarial examples. We demonstrate that robustness is improved under a variety of adversarial attacks by applying SP loss. Moreover, robust boundary learned by SP loss also performs well on imbalanced datasets.