Functional Risk Minimization
This addresses a foundational problem in machine learning by offering a new paradigm that could enhance generalization in over-parameterized regimes, though it appears incremental as it builds on ERM.
The paper tackles the limitation of Empirical Risk Minimization (ERM) by proposing Functional Risk Minimization (FRM), a framework that compares functions instead of outputs, resulting in improved performance across supervised, unsupervised, and reinforcement learning experiments.
The field of Machine Learning has changed significantly since the 1970s. However, its most basic principle, Empirical Risk Minimization (ERM), remains unchanged. We propose Functional Risk Minimization~(FRM), a general framework where losses compare functions rather than outputs. This results in better performance in supervised, unsupervised, and RL experiments. In the FRM paradigm, for each data point $(x_i,y_i)$ there is function $f_{θ_i}$ that fits it: $y_i = f_{θ_i}(x_i)$. This allows FRM to subsume ERM for many common loss functions and to capture more realistic noise processes. We also show that FRM provides an avenue towards understanding generalization in the modern over-parameterized regime, as its objective can be framed as finding the simplest model that fits the training data.