Understanding Progressive Training Through the Framework of Randomized Coordinate Descent
This work addresses the theoretical gap in Progressive Training methods for researchers and practitioners in machine learning, offering a novel algorithm with proven convergence, though it is incremental as it builds on existing PT and RCD frameworks.
The authors tackled the lack of theoretical guarantees for Progressive Training (PT) by proposing the Randomized Progressive Training (RPT) algorithm, which provides rigorous convergence analysis for general smooth objective functions and is validated through computational experiments.
We propose a Randomized Progressive Training algorithm (RPT) -- a stochastic proxy for the well-known Progressive Training method (PT) (Karras et al., 2017). Originally designed to train GANs (Goodfellow et al., 2014), PT was proposed as a heuristic, with no convergence analysis even for the simplest objective functions. On the contrary, to the best of our knowledge, RPT is the first PT-type algorithm with rigorous and sound theoretical guarantees for general smooth objective functions. We cast our method into the established framework of Randomized Coordinate Descent (RCD) (Nesterov, 2012; Richtárik & Takáč, 2014), for which (as a by-product of our investigations) we also propose a novel, simple and general convergence analysis encapsulating strongly-convex, convex and nonconvex objectives. We then use this framework to establish a convergence theory for RPT. Finally, we validate the effectiveness of our method through extensive computational experiments.