Accelerating Optimization via Differentiable Stopping Time
This addresses the challenge of efficiently tuning optimization hyperparameters for machine learning practitioners, though it is incremental as it builds on existing differentiable frameworks.
The paper tackled the problem of accelerating optimization algorithms by making the time to reach a target loss differentiable, enabling gradient-based tuning of hyperparameters. The proposed method showed superior performance in experiments across various problems.
Optimization is an important module of modern machine learning applications. Tremendous efforts have been made to accelerate optimization algorithms. A common formulation is achieving a lower loss at a given time. This enables a differentiable framework with respect to the algorithm hyperparameters. In contrast, its dual, minimizing the time to reach a target loss, is believed to be non-differentiable, as the time is not differentiable. As a result, it usually serves as a conceptual framework or is optimized using zeroth-order methods. To address this limitation, we propose a differentiable stopping time and theoretically justify it based on differential equations. An efficient algorithm is designed to backpropagate through it. As a result, the proposed differentiable stopping time enables a new differentiable formulation for accelerating algorithms. We further discuss its applications, such as online hyperparameter tuning and learning to optimize. Our proposed methods show superior performance in comprehensive experiments across various problems, which confirms their effectiveness.