Enhancing Classifier Conservativeness and Robustness by Polynomiality
This work addresses robustness issues in machine learning classifiers, particularly for adversarial attacks, though it appears incremental as it builds on existing methods like softmax and LDA.
The paper tackles the problem of overconfident decisions in classifiers like LDA and logistic regression by introducing polynomiality, which leads to random-level performance in data tails and results in softRmax, a robust alternative to softmax that improves adversarial robustness without gradient obfuscation.
We illustrate the detrimental effect, such as overconfident decisions, that exponential behavior can have in methods like classical LDA and logistic regression. We then show how polynomiality can remedy the situation. This, among others, leads purposefully to random-level performance in the tails, away from the bulk of the training data. A directly related, simple, yet important technical novelty we subsequently present is softRmax: a reasoned alternative to the standard softmax function employed in contemporary (deep) neural networks. It is derived through linking the standard softmax to Gaussian class-conditional models, as employed in LDA, and replacing those by a polynomial alternative. We show that two aspects of softRmax, conservativeness and inherent gradient regularization, lead to robustness against adversarial attacks without gradient obfuscation.