Robust Empirical Risk Minimization with Tolerance
This work addresses the need for simple and sample-efficient robust learning algorithms, offering a theoretical advance over previous exponential sample complexity results, though it is incremental as it builds on a recent relaxation of the robust model.
The paper tackles the problem of robust classification by showing that a tolerant variant of empirical risk minimization can learn VC classes under geometric conditions with sample complexity that is polynomial in relevant parameters, specifically requiring only O(VC(H)d log(D/(γδ))/ε²) samples.
Developing simple, sample-efficient learning algorithms for robust classification is a pressing issue in today's tech-dominated world, and current theoretical techniques requiring exponential sample complexity and complicated improper learning rules fall far from answering the need. In this work we study the fundamental paradigm of (robust) $\textit{empirical risk minimization}$ (RERM), a simple process in which the learner outputs any hypothesis minimizing its training error. RERM famously fails to robustly learn VC classes (Montasser et al., 2019a), a bound we show extends even to `nice' settings such as (bounded) halfspaces. As such, we study a recent relaxation of the robust model called $\textit{tolerant}$ robust learning (Ashtiani et al., 2022) where the output classifier is compared to the best achievable error over slightly larger perturbation sets. We show that under geometric niceness conditions, a natural tolerant variant of RERM is indeed sufficient for $γ$-tolerant robust learning VC classes over $\mathbb{R}^d$, and requires only $\tilde{O}\left( \frac{VC(H)d\log \frac{D}{γδ}}{ε^2}\right)$ samples for robustness regions of (maximum) diameter $D$.