A Tunable Loss Function for Robust Classification: Calibration, Landscape, and Generalization
This work addresses robustness and class imbalance issues in classification tasks for practitioners, offering a tunable loss function with theoretical grounding, though it is incremental as it builds upon existing loss functions.
The authors tackled the problem of improving classification robustness and handling class imbalance by introducing a tunable loss function called α-loss, which interpolates between exponential, log, and 0-1 losses, and demonstrated that tuning α can enhance model robustness to label flips and sensitivity to imbalanced classes in experiments on benchmark image datasets.
We introduce a tunable loss function called $α$-loss, parameterized by $α\in (0,\infty]$, which interpolates between the exponential loss ($α= 1/2$), the log-loss ($α= 1$), and the 0-1 loss ($α= \infty$), for the machine learning setting of classification. Theoretically, we illustrate a fundamental connection between $α$-loss and Arimoto conditional entropy, verify the classification-calibration of $α$-loss in order to demonstrate asymptotic optimality via Rademacher complexity generalization techniques, and build-upon a notion called strictly local quasi-convexity in order to quantitatively characterize the optimization landscape of $α$-loss. Practically, we perform class imbalance, robustness, and classification experiments on benchmark image datasets using convolutional-neural-networks. Our main practical conclusion is that certain tasks may benefit from tuning $α$-loss away from log-loss ($α= 1$), and to this end we provide simple heuristics for the practitioner. In particular, navigating the $α$ hyperparameter can readily provide superior model robustness to label flips ($α> 1$) and sensitivity to imbalanced classes ($α< 1$).