Infinite Limits of Multi-head Transformer Dynamics
This work provides theoretical insights into transformer scaling for researchers in machine learning, but it is incremental as it builds on existing analysis of neural network dynamics.
The authors analyzed scaling limits of transformer training dynamics in the feature learning regime, identifying parameterizations that allow well-defined infinite width and depth limits and using dynamical mean field theory to describe different infinite limits, with numerical evidence of convergence.
In this work, we analyze various scaling limits of the training dynamics of transformer models in the feature learning regime. We identify the set of parameterizations that admit well-defined infinite width and depth limits, allowing the attention layers to update throughout training--a relevant notion of feature learning in these models. We then use tools from dynamical mean field theory (DMFT) to analyze various infinite limits (infinite key/query dimension, infinite heads, and infinite depth) which have different statistical descriptions depending on which infinite limit is taken and how attention layers are scaled. We provide numerical evidence of convergence to the limits and discuss how the parameterization qualitatively influences learned features.