High dimensional theory of two-phase optimizers
For machine learning practitioners, this work provides theoretical insights into two-phase optimizers, potentially improving large-scale training efficiency.
The paper analyzes LA-DiLoCo, a two-phase optimizer, on high-dimensional linear regression, showing that its one-worker variant (LA) offers a beneficial signal-noise tradeoff over SGD, and that multi-worker noise can be mitigated via hyperparameter tuning. It also shows that SLA (LA with momentum) can accelerate training via a nonlinear transformation of the Hessian spectrum, maximized with Nesterov momentum.
The trend towards larger training setups has brought a renewed interest in partially asynchronous two-phase optimizers which optimize locally and then synchronize across workers. Additionally, recent work suggests that the one-worker version of one of these algorithms, DiLoCo, shows promising results as a (synchronous) optimizer. Motivated by these studies we present an analysis of LA-DiLoCo, a simple member of the DiLoCo family, on a high-dimensional linear regression problem. We show that the one-worker variant, LA, provides a different tradeoff between signal and noise than SGD, which is beneficial in many scenarios. We also show that the multi-worker version generates more noise than the single worker version, but that this additional noise generation can be ameliorated by appropriate choice of hyperparameters. We conclude with an analysis of SLA -- LA with momentum -- and show that stacking two momentum operators gives an opportunity for acceleration via a non-linear transformation of the "effective'' Hessian spectrum, which is maximized for Nesterov momentum. Altogether our results show that two-phase optimizers represent a fruitful new paradigm for understanding and improving training algorithms.