Incremental Learning in Mirror Flows
Provides a theoretical mechanism for incremental learning in mirror descent, relevant for understanding optimization dynamics in machine learning.
The paper shows that mirror flows with convex quadratic loss and general convex mirror potential exhibit incremental learning when initialized near the domain boundary, with rescaled trajectories converging to a limiting flow where the primal variable minimizes loss over a time-dependent hypothesis set.
We study mirror flows generated by a convex quadratic loss and a general convex lower semicontinuous mirror potential. We show that, when initialized near the boundary of the domain of the mirror potential, their rescaled trajectories converge to a limiting mirror flow whose potential is the indicator function of the domain. In this limit, the primal variable minimizes the loss over a time-dependent hypothesis set: the subdifferential of the support function of the domain, evaluated at the dual variable. This characterization provides a general mechanism for incremental learning in mirror flows.