Improving Energy Conserving Descent for Machine Learning: Theory and Practice
This work addresses optimization challenges for machine learning practitioners, offering a novel framework with potential for broader application, though it appears incremental in improving upon existing ECD realizations.
The paper tackles optimization in machine learning by developing the theory of Energy Conserving Descent (ECD) and introducing ECDSep, a gradient-based algorithm for convex and non-convex problems, achieving competitive or improved performance compared to methods like SGD, Adam, and AdamW across various tasks.
We develop the theory of Energy Conserving Descent (ECD) and introduce ECDSep, a gradient-based optimization algorithm able to tackle convex and non-convex optimization problems. The method is based on the novel ECD framework of optimization as physical evolution of a suitable chaotic energy-conserving dynamical system, enabling analytic control of the distribution of results - dominated at low loss - even for generic high-dimensional problems with no symmetries. Compared to previous realizations of this idea, we exploit the theoretical control to improve both the dynamics and chaos-inducing elements, enhancing performance while simplifying the hyper-parameter tuning of the optimization algorithm targeted to different classes of problems. We empirically compare with popular optimization methods such as SGD, Adam and AdamW on a wide range of machine learning problems, finding competitive or improved performance compared to the best among them on each task. We identify limitations in our analysis pointing to possibilities for additional improvements.