On the connections between algorithmic regularization and penalization for convex losses
This work provides a theoretical foundation for understanding regularization in optimization, which is incremental but clarifies connections in convex settings.
The paper tackles the problem of connecting algorithmic regularization and explicit penalization for convex losses, establishing their equivalence under a geometric condition on the optimization path.
In this work we establish the equivalence of algorithmic regularization and explicit convex penalization for generic convex losses. We introduce a geometric condition for the optimization path of a convex function, and show that if such a condition is satisfied, the optimization path of an iterative algorithm on the unregularized optimization problem can be represented as the solution path of a corresponding penalized problem.