Meta-Principled Family of Hyperparameter Scaling Strategies
This work addresses the problem of scaling neural networks effectively for researchers and practitioners, though it appears incremental as it builds on existing scaling theories.
The paper derives a one-parameter family of hyperparameter scaling strategies that interpolates between neural-tangent and mean-field/maximal-update scaling, revealing how to scale depth with width to maintain representation-learning ability in large-scale models.
In this note, we first derive a one-parameter family of hyperparameter scaling strategies that interpolates between the neural-tangent scaling and mean-field/maximal-update scaling. We then calculate the scalings of dynamical observables -- network outputs, neural tangent kernels, and differentials of neural tangent kernels -- for wide and deep neural networks. These calculations in turn reveal a proper way to scale depth with width such that resultant large-scale models maintain their representation-learning ability. Finally, we observe that various infinite-width limits examined in the literature correspond to the distinct corners of the interconnected web spanned by effective theories for finite-width neural networks, with their training dynamics ranging from being weakly-coupled to being strongly-coupled.