AdaCubic: An Adaptive Cubic Regularization Optimizer for Deep Learning
This addresses the need for efficient and easy-to-use optimizers in deep learning, particularly in settings where hyperparameter fine-tuning is infeasible, though it appears incremental as it builds on existing cubic regularization methods.
The paper tackles the problem of optimizing deep learning models by proposing AdaCubic, an adaptive cubic regularization optimizer that dynamically adjusts the cubic term weight, and it demonstrates competitive or superior performance compared to existing optimizers in tasks across Computer Vision, Natural Language Processing, and Signal Processing, with a fixed set of hyperparameters.
A novel regularization technique, AdaCubic, is proposed that adapts the weight of the cubic term. The heart of AdaCubic is an auxiliary optimization problem with cubic constraints that dynamically adjusts the weight of the cubic term in Newton's cubic regularized method. We use Hutchinson's method to approximate the Hessian matrix, thereby reducing computational cost. We demonstrate that AdaCubic inherits the cubically regularized Newton method's local convergence guarantees. Our experiments in Computer Vision, Natural Language Processing, and Signal Processing tasks demonstrate that AdaCubic outperforms or competes with several widely used optimizers. Unlike other adaptive algorithms that require hyperparameter fine-tuning, AdaCubic is evaluated with a fixed set of hyperparameters, rendering it a highly attractive optimizer in settings where fine-tuning is infeasible. This makes AdaCubic an attractive option for researchers and practitioners alike. To our knowledge, AdaCubic is the first optimizer to leverage cubic regularization in scalable deep learning applications.