Robust Linear Predictions: Analyses of Uniform Concentration, Fast Rates and Model Misspecification
This provides a robust solution for linear prediction problems in statistics and machine learning, addressing outliers and model misspecification, but it is incremental as it builds on existing Median of Means approaches.
The paper tackles robust linear prediction by developing a unified framework for Hilbert spaces with generic loss functions, achieving an error rate of O(max{|O|^{1/2}n^{-1/2}, |I|^{1/2}n^{-1}} + ε) under mild conditions, which matches best-known rates but is slower than classical rates.
The problem of linear predictions has been extensively studied for the past century under pretty generalized frameworks. Recent advances in the robust statistics literature allow us to analyze robust versions of classical linear models through the prism of Median of Means (MoM). Combining these approaches in a piecemeal way might lead to ad-hoc procedures, and the restricted theoretical conclusions that underpin each individual contribution may no longer be valid. To meet these challenges coherently, in this study, we offer a unified robust framework that includes a broad variety of linear prediction problems on a Hilbert space, coupled with a generic class of loss functions. Notably, we do not require any assumptions on the distribution of the outlying data points ($\mathcal{O}$) nor the compactness of the support of the inlying ones ($\mathcal{I}$). Under mild conditions on the dual norm, we show that for misspecification level $ε$, these estimators achieve an error rate of $O(\max\left\{|\mathcal{O}|^{1/2}n^{-1/2}, |\mathcal{I}|^{1/2}n^{-1} \right\}+ε)$, matching the best-known rates in literature. This rate is slightly slower than the classical rates of $O(n^{-1/2})$, indicating that we need to pay a price in terms of error rates to obtain robust estimates. Additionally, we show that this rate can be improved to achieve so-called "fast rates" under additional assumptions.