A Trainable Optimizer
This work addresses optimization efficiency for machine learning practitioners, offering a novel method that reduces variance with minimal computational overhead.
The paper tackles the problem of manual gradient estimation in optimization by introducing a trainable optimizer framework that jointly trains gradient estimators and model weights, achieving faster convergence than benchmark algorithms like ADAM in strongly convex, non-convex, and LLM fine-tuning settings.
The concept of learning to optimize involves utilizing a trainable optimization strategy rather than relying on manually defined full gradient estimations such as ADAM. We present a framework that jointly trains the full gradient estimator and the trainable weights of the model. Specifically, we prove that pseudo-linear TO (Trainable Optimizer), a linear approximation of the full gradient, matches SGD's convergence rate while effectively reducing variance. Pseudo-linear TO incurs negligible computational overhead, requiring only minimal additional tensor multiplications. To further improve computational efficiency, we introduce two simplified variants of Pseudo-linear TO. Experiments demonstrate that TO methods converge faster than benchmark algorithms (e.g., ADAM) in both strongly convex and non-convex settings, and fine tuning of an LLM.