A Dual Optimization View to Empirical Risk Minimization with f-Divergence Regularization
This work addresses a theoretical and computational challenge in machine learning optimization, but it appears incremental as it builds on existing dual optimization and regularization frameworks.
The paper tackles the problem of empirical risk minimization with f-divergence regularization by introducing a dual formulation, connecting it to a normalization function via a nonlinear ODE, and providing a computationally efficient method to calculate this function under mild conditions.
The dual formulation of empirical risk minimization with f-divergence regularization (ERM-fDR) is introduced. The solution of the dual optimization problem to the ERM-fDR is connected to the notion of normalization function introduced as an implicit function. This dual approach leverages the Legendre-Fenchel transform and the implicit function theorem to provide a nonlinear ODE expression to the normalization function. Furthermore, the nonlinear ODE expression and its properties provide a computationally efficient method to calculate the normalization function of the ERM-fDR solution under a mild condition.