Equivalence of the Empirical Risk Minimization to Regularization on the Family of f-Divergences
This work provides a theoretical framework for regularization in machine learning, offering insights into inductive biases and equivalence relations, but it is incremental as it builds on known f-divergence concepts.
The paper tackles the problem of empirical risk minimization with f-divergence regularization by presenting a solution under mild conditions, showing uniqueness of the optimal measure and deriving properties such as support alignment with the reference measure and equivalence transformations of the risk function.
The solution to empirical risk minimization with $f$-divergence regularization (ERM-$f$DR) is presented under mild conditions on $f$. Under such conditions, the optimal measure is shown to be unique. Examples of the solution for particular choices of the function $f$ are presented. Previously known solutions to common regularization choices are obtained by leveraging the flexibility of the family of $f$-divergences. These include the unique solutions to empirical risk minimization with relative entropy regularization (Type-I and Type-II). The analysis of the solution unveils the following properties of $f$-divergences when used in the ERM-$f$DR problem: $i\bigl)$ $f$-divergence regularization forces the support of the solution to coincide with the support of the reference measure, which introduces a strong inductive bias that dominates the evidence provided by the training data; and $ii\bigl)$ any $f$-divergence regularization is equivalent to a different $f$-divergence regularization with an appropriate transformation of the empirical risk function.