Direct From Darwin: Deriving Advanced Optimizers From Evolutionary First Principles
This work provides a theoretical unification of evolutionary biology and optimization, offering a new lens for understanding and potentially improving gradient-based optimizers, but the practical impact is currently incremental as it does not demonstrate performance gains.
The authors derive a family of gradient-based optimizers (SGD, Natural Gradient Descent, Adam, etc.) from evolutionary first principles by introducing Darwinian Lineage Simulations (DLS) noise, proving that these algorithms become faithful simulations of Darwinian evolution. They unify Fisher's and Wright's evolutionary theories and show that adding DLS noise makes existing optimizers evolutionarily valid.
Evolutionary computation has long promised to deliver both high-performance optimization tools as well as rigorous scientific simulations of Darwinian evolution. However, modern algorithms frequently abandon evolutionary fidelity for physics-inspired heuristics or superficial biological metaphors. This paper derives a suite of advanced gradient-based optimization algorithms directly from evolutionary first principles. We introduce Darwinian Lineage Simulations (DLS) to prove that, in an asexual context, Fisher's and Wright's historically opposed views of evolution are actually formally equivalent. This unification requires carefully partitioning Fisher's deterministically-evolving total population into Wright's randomly-drifting sub-populations. We prove that proper bookkeeping requires introducing a specific kind of structured noise (the DLS noise relation). Crucially, however, any bookkeeping choices which satisfy this relation will result in a faithful simulation of evolution. Using this vast representational freedom, we prove that a broad family of battle-tested optimization algorithms are already perfectly compatible with evolutionary dynamics. These include: Stochastic Gradient Descent, Natural Gradient Descent, and the Damped Newton's method among many others. By simply adding DLS noise (i.e., evolutionarily faithful genetic drift), these algorithms become scientifically valid in silico simulations of Darwinian evolution. Finally, we demonstrate that even the state-of-the-art Adam optimizer can be brought into evolutionary compliance through a minor mathematical surgery.