A Split-Client Approach to Second-Order Optimization
This work addresses a bottleneck in optimization for machine learning practitioners by enabling more efficient use of second-order methods, though it is incremental in its approach.
The paper tackles the high computational cost of second-order optimization methods by proposing a split-client framework that computes gradients and curvature asynchronously, achieving a provable wall-clock speedup of order √τ, with experiments showing consistent outperformance over baselines.
Second-order methods promise faster convergence but are rarely used in practice because Hessian computations and decompositions are far more expensive than gradients. We propose a \emph{split-client} framework where gradients and curvature are computed asynchronously by separate clients. This abstraction captures realistic delays and inexact Hessian updates while avoiding the manual tuning required by Lazy Hessian methods. Focusing on cubic regularization, we show that our approach retains strong convergence guarantees and achieves a provable wall-clock speedup of order $\sqrtτ$, where $τ$ is the relative time needed to compute and decompose the Hessian compared to a gradient step. Since $τ$ can be orders of magnitude larger than one in high-dimensional problems, this improvement is practically significant. Experiments on synthetic and real datasets confirm the theory: asynchronous curvature consistently outperforms vanilla and Lazy Hessian baselines, while maintaining second-order accuracy.