Smoothly Giving up: Robustness for Simple Models
It addresses robustness for interpretable and low-cost models in domains like health care and federated learning, but is incremental as it builds on existing loss functions.
The paper tackles the problem of label noise in simple models like logistic regression and boosting by using the margin-based α-loss to robustly train them, showing efficacy on the Long-Servedio and COVID-19 datasets.
There is a growing need for models that are interpretable and have reduced energy and computational cost (e.g., in health care analytics and federated learning). Examples of algorithms to train such models include logistic regression and boosting. However, one challenge facing these algorithms is that they provably suffer from label noise; this has been attributed to the joint interaction between oft-used convex loss functions and simpler hypothesis classes, resulting in too much emphasis being placed on outliers. In this work, we use the margin-based $α$-loss, which continuously tunes between canonical convex and quasi-convex losses, to robustly train simple models. We show that the $α$ hyperparameter smoothly introduces non-convexity and offers the benefit of "giving up" on noisy training examples. We also provide results on the Long-Servedio dataset for boosting and a COVID-19 survey dataset for logistic regression, highlighting the efficacy of our approach across multiple relevant domains.