A Researcher's Guide to Empirical Risk Minimization
This work provides a theoretical framework for analyzing ERM, which is incremental as it organizes and extends existing derivations to handle complex scenarios like nuisance estimation, benefiting researchers in statistical learning and causal inference.
The paper develops high-probability regret bounds for empirical risk minimization (ERM) using a modular approach based on localized Rademacher complexity, recovering rates for various function classes like VC-subgraph and Sobolev/Hölder, and extends this to ERM with nuisance components in applications such as causal inference and domain adaptation, deriving regret-transfer bounds that decompose error into statistical and approximation terms.
This guide develops high-probability regret bounds for empirical risk minimization (ERM). The presentation is modular: we state broadly applicable guarantees under high-level conditions and give tools for verifying them for specific losses and function classes. We emphasize that many ERM rate derivations can be organized around a three-step recipe -- a basic inequality, a uniform local concentration bound, and a fixed-point argument -- which yields regret bounds in terms of a critical radius, defined via localized Rademacher complexity, under a mild Bernstein-type variance--risk condition. To make these bounds concrete, we upper bound the critical radius using local maximal inequalities and metric-entropy integrals, recovering familiar rates for VC-subgraph, Sobolev/Hölder, and bounded-variation classes. We also review ERM with nuisance components -- including weighted ERM and Neyman-orthogonal losses -- as they arise in causal inference, missing data, and domain adaptation. Following the orthogonal learning framework, we highlight that these problems often admit regret-transfer bounds linking regret under an estimated loss to population regret under the target loss. These bounds typically decompose regret into (i) statistical error under the estimated (optimized) loss and (ii) approximation error due to nuisance estimation. Under sample splitting or cross-fitting, the first term can be controlled using standard fixed-loss ERM regret bounds, while the second term depends only on nuisance-estimation accuracy. We also treat the in-sample regime, where nuisances and the ERM are fit on the same data, deriving regret bounds and giving sufficient conditions for fast rates.