Optimal Learning Rate Scaling Depends on Data in Deep Scalar Linear Networks
For researchers studying learning rate scaling in deep learning, this work highlights the importance of data-dependent scaling even in minimal models, challenging the assumption of data-agnostic rules.
The paper shows that in deep scalar linear networks, the optimal learning rate scaling depends on data, and data-agnostic scaling fails across depths. Under data-dependent scaling, learning dynamics become independent of data and weakly dependent on depth, achieving constant linear convergence for all depths.
In this short note we consider the gradient descent dynamics of deep scalar linear networks, $f(x) = \prod_{l=1}^L w_l x$, which enjoy exact time-course solutions for any integer depth. We show that even in this minimal model, the optimal depth-wise learning rate scaling depends on data, whereas data-agnostic scaling rules fail to transfer across depths. Under the data-dependent optimal scaling, the learning dynamics is independent of data and weakly dependent on depth, resulting in a constant linear convergence rate across all depths including infinity. We further show similar data-dependent effects in deep scalar linear networks with residual connections.