Probabilistically Robust Learning: Balancing Average- and Worst-case Performance
This addresses robustness issues in safety-critical domains by offering a less conservative alternative to adversarial training, though it is incremental as it builds on existing robustness paradigms.
The paper tackles the trade-off between accuracy and robustness in machine learning by proposing a probabilistic robustness framework that enforces robustness to most perturbations rather than all, balancing average- and worst-case performance with lower computational cost than adversarial training, achieving competitive results on datasets like MNIST, CIFAR-10, and SVHN.
Many of the successes of machine learning are based on minimizing an averaged loss function. However, it is well-known that this paradigm suffers from robustness issues that hinder its applicability in safety-critical domains. These issues are often addressed by training against worst-case perturbations of data, a technique known as adversarial training. Although empirically effective, adversarial training can be overly conservative, leading to unfavorable trade-offs between nominal performance and robustness. To this end, in this paper we propose a framework called probabilistic robustness that bridges the gap between the accurate, yet brittle average case and the robust, yet conservative worst case by enforcing robustness to most rather than to all perturbations. From a theoretical point of view, this framework overcomes the trade-offs between the performance and the sample-complexity of worst-case and average-case learning. From a practical point of view, we propose a novel algorithm based on risk-aware optimization that effectively balances average- and worst-case performance at a considerably lower computational cost relative to adversarial training. Our results on MNIST, CIFAR-10, and SVHN illustrate the advantages of this framework on the spectrum from average- to worst-case robustness.