Second-Order, First-Class: A Composable Stack for Curvature-Aware Training
This work addresses a system-level bottleneck for researchers and practitioners in machine learning by providing a more efficient and flexible framework for second-order optimization, though it is incremental as it builds on existing optimization libraries like Optax.
The authors tackled the underuse of second-order methods in machine learning due to implementation complexity and lack of composability by introducing Somax, a composable stack for curvature-aware training, which reduces per-step overhead and improves scaling behavior through static planning and explicit module choices.
Second-order methods promise improved stability and faster convergence, yet they remain underused due to implementation overhead, tuning brittleness, and the lack of composable APIs. We introduce Somax, a composable Optax-native stack that treats curvature-aware training as a single JIT-compiled step governed by a static plan. Somax exposes first-class modules -- curvature operators, estimators, linear solvers, preconditioners, and damping policies -- behind a single step interface and composes with Optax by applying standard gradient transformations (e.g., momentum, weight decay, schedules) to the computed direction. This design makes typically hidden choices explicit and swappable. Somax separates planning from execution: it derives a static plan (including cadences) from module requirements, then runs the step through a specialized execution path that reuses intermediate results across modules. We report system-oriented ablations showing that (i) composition choices materially affect scaling behavior and time-to-accuracy, and (ii) planning reduces per-step overhead relative to unplanned composition with redundant recomputation.