Learning without Concentration for General Loss Functions
This work addresses robust learning under heavy-tailed distributions for researchers in statistical learning theory, offering a framework to handle outliers and non-concentration, though it appears incremental in extending existing risk minimization methods.
The paper tackles prediction and estimation problems using empirical risk minimization with general convex loss functions, achieving sharp error rates even in heavy-tailed scenarios where concentration fails. The results show error rates depend on intrinsic class complexity and interactions between class members, target, and loss, with the latter dominating in non-realizable problems.
We study prediction and estimation problems using empirical risk minimization, relative to a general convex loss function. We obtain sharp error rates even when concentration is false or is very restricted, for example, in heavy-tailed scenarios. Our results show that the error rate depends on two parameters: one captures the intrinsic complexity of the class, and essentially leads to the error rate in a noise-free (or realizable) problem; the other measures interactions between class members the target and the loss, and is dominant when the problem is far from realizable. We also explain how one may deal with outliers by choosing the loss in a way that is calibrated to the intrinsic complexity of the class and to the noise-level of the problem (the latter is measured by the distance between the target and the class).