Benign Overfitting under Learning Rate Conditions for $α$ Sub-exponential Input
This work provides theoretical insights into benign overfitting for heavy-tailed data, which is incremental as it generalizes prior sub-gaussian results to more realistic data environments.
The paper investigates benign overfitting in binary classification with heavy-tailed input distributions, extending analysis to α sub-exponential distributions and showing that the misclassification error asymptotically approaches the noise level under certain conditions on dimensionality and distribution centers.
This paper investigates the phenomenon of benign overfitting in binary classification problems with heavy-tailed input distributions, extending the analysis of maximum margin classifiers to $α$ sub-exponential distributions ($α\in (0, 2]$). This generalizes previous work focused on sub-gaussian inputs. We provide generalization error bounds for linear classifiers trained using gradient descent on unregularized logistic loss in this heavy-tailed setting. Our results show that, under certain conditions on the dimensionality $p$ and the distance between the centers of the distributions, the misclassification error of the maximum margin classifier asymptotically approaches the noise level, the theoretical optimal value. Moreover, we derive an upper bound on the learning rate $β$ for benign overfitting to occur and show that as the tail heaviness of the input distribution $α$ increases, the upper bound on the learning rate decreases. These results demonstrate that benign overfitting persists even in settings with heavier-tailed inputs than previously studied, contributing to a deeper understanding of the phenomenon in more realistic data environments.