Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit
This work addresses the rising cost of hyperparameter tuning in deep learning for practitioners by enabling transfer across network dimensions, though it is incremental as it builds on existing μP methods.
The paper tackles the problem of hyperparameter tuning across network depth by proposing a residual network parameterization with a residual branch scale of 1/√depth combined with μP, demonstrating that optimal hyperparameters transfer across both width and depth on datasets like CIFAR-10 and ImageNet, supported by theoretical analysis using dynamical mean field theory.
The cost of hyperparameter tuning in deep learning has been rising with model sizes, prompting practitioners to find new tuning methods using a proxy of smaller networks. One such proposal uses $μ$P parameterized networks, where the optimal hyperparameters for small width networks transfer to networks with arbitrarily large width. However, in this scheme, hyperparameters do not transfer across depths. As a remedy, we study residual networks with a residual branch scale of $1/\sqrt{\text{depth}}$ in combination with the $μ$P parameterization. We provide experiments demonstrating that residual architectures including convolutional ResNets and Vision Transformers trained with this parameterization exhibit transfer of optimal hyperparameters across width and depth on CIFAR-10 and ImageNet. Furthermore, our empirical findings are supported and motivated by theory. Using recent developments in the dynamical mean field theory (DMFT) description of neural network learning dynamics, we show that this parameterization of ResNets admits a well-defined feature learning joint infinite-width and infinite-depth limit and show convergence of finite-size network dynamics towards this limit.