Learning-Rate-Free Learning: Dissecting D-Adaptation and Probabilistic Line Search
This is an incremental analysis for researchers in optimization, focusing on comparing and potentially combining existing techniques without introducing novel advancements.
The paper examines D-Adaptation and probabilistic line search, two methods for automating learning rate selection in stochastic gradient descent to avoid manual tuning, but does not present new experimental results or numerical improvements.
This paper explores two recent methods for learning rate optimisation in stochastic gradient descent: D-Adaptation (arXiv:2301.07733) and probabilistic line search (arXiv:1502.02846). These approaches aim to alleviate the burden of selecting an initial learning rate by incorporating distance metrics and Gaussian process posterior estimates, respectively. In this report, I provide an intuitive overview of both methods, discuss their shared design goals, and devise scope for merging the two algorithms.