The Feature Speed Formula: a flexible approach to scale hyper-parameters of deep neural networks
This addresses the challenge of indirect control over feature learning in deep learning for researchers and practitioners, offering a flexible approach to hyper-parameter scaling, though it is incremental as it builds on prior work.
The paper tackles the problem of tuning hyper-parameters in deep neural networks by introducing a feature speed formula based on the angle between feature updates and the backward pass, showing that it can predict and control feature learning, with results including a new hyper-parameter scaling for ReLU MLPs that maintains favorable theoretical properties.
Deep learning succeeds by doing hierarchical feature learning, yet tuning hyper-parameters (HP) such as initialization scales, learning rates etc., only give indirect control over this behavior. In this paper, we introduce a key notion to predict and control feature learning: the angle $θ_\ell$ between the feature updates and the backward pass (at layer index $\ell$). We show that the magnitude of feature updates after one GD step, at any training time, can be expressed via a simple and general \emph{feature speed formula} in terms of this angle $θ_\ell$, the loss decay, and the magnitude of the backward pass. This angle $θ_\ell$ is controlled by the conditioning of the layer-to-layer Jacobians and at random initialization, it is determined by the spectrum of a certain kernel, which coincides with the Neural Tangent Kernel when $\ell=\text{depth}$. Given $θ_\ell$, the feature speed formula provides us with rules to adjust HPs (scales and learning rates) so as to satisfy certain dynamical properties, such as feature learning and loss decay. We investigate the implications of our approach for ReLU MLPs and ResNets in the large width-then-depth limit. Relying on prior work, we show that in ReLU MLPs with iid initialization, the angle degenerates with depth as $\cos(θ_\ell)=Θ(1/\sqrt{\ell})$. In contrast, ResNets with branch scale $O(1/\sqrt{\text{depth}})$ maintain a non-degenerate angle $\cos(θ_\ell)=Θ(1)$. We use these insights to recover key properties of known HP scalings and also to introduce a new HP scaling for large depth ReLU MLPs with favorable theoretical properties.