Control Interpretations for First-Order Optimization Methods
For researchers in optimization and machine learning, this work provides a novel perspective that bridges control theory and optimization, potentially enabling new algorithm designs.
This paper interprets first-order optimization methods through control theory, showing they consist of basic control elements like PID and lag compensators, and uses the small gain theorem to link convergence rate analysis to input-output gain computations, suggesting control synthesis tools can design optimization algorithms.
First-order iterative optimization methods play a fundamental role in large scale optimization and machine learning. This paper presents control interpretations for such optimization methods. First, we give loop-shaping interpretations for several existing optimization methods and show that they are composed of basic control elements such as PID and lag compensators. Next, we apply the small gain theorem to draw a connection between the convergence rate analysis of optimization methods and the input-output gain computations of certain complementary sensitivity functions. These connections suggest that standard classical control synthesis tools may be brought to bear on the design of optimization algorithms.