Gradient Descent as Loss Landscape Navigation: a Normative Framework for Deriving Learning Rules
This provides a principled foundation for designing adaptive algorithms in machine learning, though it is incremental as it unifies existing rules rather than introducing a new paradigm.
The paper tackles the problem of understanding why learning rules work by proposing a theoretical framework that casts them as policies for navigating loss landscapes, showing that optimal rules emerge from an optimal control problem and unifying phenomena like gradient descent, momentum, and adaptive optimizers under a single objective.
Learning rules -- prescriptions for updating model parameters to improve performance -- are typically assumed rather than derived. Why do some learning rules work better than others, and under what assumptions can a given rule be considered optimal? We propose a theoretical framework that casts learning rules as policies for navigating (partially observable) loss landscapes, and identifies optimal rules as solutions to an associated optimal control problem. A range of well-known rules emerge naturally within this framework under different assumptions: gradient descent from short-horizon optimization, momentum from longer-horizon planning, natural gradients from accounting for parameter space geometry, non-gradient rules from partial controllability, and adaptive optimizers like Adam from online Bayesian inference of loss landscape shape. We further show that continual learning strategies like weight resetting can be understood as optimal responses to task uncertainty. By unifying these phenomena under a single objective, our framework clarifies the computational structure of learning and offers a principled foundation for designing adaptive algorithms.