Fast Compute for ML Optimization
This addresses optimization efficiency for machine learning practitioners by offering a method that automates hyperparameter tuning, though it appears incremental as it builds on existing EM and Nesterov concepts.
The paper tackles the problem of removing user-specified learning-rate and momentum schedules in optimization by proposing the Scale Mixture EM (SM-EM) algorithm, which achieves up to 13× lower final loss than Adam on synthetic benchmarks and a 10× runtime reduction for regularization paths.
We study optimization for losses that admit a variance-mean scale-mixture representation. Under this representation, each EM iteration is a weighted least squares update in which latent variables determine observation and parameter weights; these play roles analogous to Adam's second-moment scaling and AdamW's weight decay, but are derived from the model. The resulting Scale Mixture EM (SM-EM) algorithm removes user-specified learning-rate and momentum schedules. On synthetic ill-conditioned logistic regression benchmarks with $p \in \{20, \ldots, 500\}$, SM-EM with Nesterov acceleration attains up to $13\times$ lower final loss than Adam tuned by learning-rate grid search. For a 40-point regularization path, sharing sufficient statistics across penalty values yields a $10\times$ runtime reduction relative to the same tuned-Adam protocol. For the base (non-accelerated) algorithm, EM monotonicity guarantees nonincreasing objective values; adding Nesterov extrapolation trades this guarantee for faster empirical convergence.