Optimality of Robust Online Learning
This provides a robust alternative for online learning in regression, addressing noise and outliers, but appears incremental as it builds on existing robust loss frameworks.
The paper tackles robust online regression by proposing an algorithm with a robust loss function in RKHS, achieving optimal capacity-independent convergence in mean square distance and, with additional information, optimal capacity-dependent rates in RKHS.
In this paper, we study an online learning algorithm with a robust loss function $\mathcal{L}_σ$ for regression over a reproducing kernel Hilbert space (RKHS). The loss function $\mathcal{L}_σ$ involving a scaling parameter $σ>0$ can cover a wide range of commonly used robust losses. The proposed algorithm is then a robust alternative for online least squares regression aiming to estimate the conditional mean function. For properly chosen $σ$ and step size, we show that the last iterate of this online algorithm can achieve optimal capacity independent convergence in the mean square distance. Moreover, if additional information on the underlying function space is known, we also establish optimal capacity dependent rates for strong convergence in RKHS. To the best of our knowledge, both of the two results are new to the existing literature of online learning.