FANoS: Friction-Adaptive Nosé--Hoover Symplectic Momentum for Stiff Objectives
This work addresses optimization challenges in machine learning for researchers, but it is incremental as it synthesizes existing ideas without broad superiority over modern baselines.
The paper tackled the problem of optimizing stiff objectives by proposing FANoS, a physics-inspired optimizer combining momentum, a thermostat variable, and a symplectic integrator, which achieved a mean final objective value of 1.74e-2 on the Rosenbrock-100D benchmark but underperformed compared to AdamW with clipping and L-BFGS.
We study a physics-inspired optimizer, \emph{FANoS} (Friction-Adaptive Nosé--Hoover Symplectic momentum), which combines (i) a momentum update written as a discretized second-order dynamical system, (ii) a Nosé--Hoover-like thermostat variable that adapts a scalar friction coefficient using kinetic-energy feedback, and (iii) a semi-implicit (symplectic-Euler) integrator, optionally with a diagonal RMS preconditioner. The method is motivated by structure-preserving integration and thermostat ideas from molecular dynamics, but is used here purely as an optimization heuristic. We provide the algorithm and limited theoretical observations in idealized settings. On the deterministic Rosenbrock-100D benchmark with 3000 gradient evaluations, FANoS-RMS attains a mean final objective value of $1.74\times 10^{-2}$, improving substantially over unclipped AdamW ($48.50$) and SGD+momentum ($90.76$) in this protocol. However, AdamW with gradient clipping is stronger, reaching $1.87\times 10^{-3}$, and L-BFGS reaches $\approx 4.4\times 10^{-10}$. On ill-conditioned convex quadratics and in a small PINN warm-start suite (Burgers and Allen--Cahn), the default FANoS configuration underperforms AdamW and can be unstable or high-variance. Overall, the evidence supports a conservative conclusion: FANoS is an interpretable synthesis of existing ideas that can help on some stiff nonconvex valleys, but it is not a generally superior replacement for modern baselines, and its behavior is sensitive to temperature-schedule and hyperparameter choices.