Derivation of Coordinate Descent Algorithms from Optimal Control Theory
This work provides a theoretical unification for optimization algorithms, which is incremental as it builds on prior propositions to derive specific methods.
The paper tackles the problem of unifying optimization algorithms by deriving coordinate descent methods from optimal control theory, showing that these algorithms can be derived using a maximum principle and control Lyapunov functions, with convergence linked to controlled dissipation and the Hessian as the operational metric.
Recently, it was posited that disparate optimization algorithms may be coalesced in terms of a central source emanating from optimal control theory. Here we further this proposition by showing how coordinate descent algorithms may be derived from this emerging new principle. In particular, we show that basic coordinate descent algorithms can be derived using a maximum principle and a collection of max functions as "control" Lyapunov functions. The convergence of the resulting coordinate descent algorithms is thus connected to the controlled dissipation of their corresponding Lyapunov functions. The operational metric for the search vector in all cases is given by the Hessian of the convex objective function.