Effective Theory of Transformers at Initialization
This work addresses hyperparameter tuning for Transformers, which is crucial for researchers and practitioners in machine learning, though it appears incremental as it builds on existing effective-theory frameworks.
The authors tackled the problem of forward-backward signal propagation in wide and deep Transformers by performing an effective-theory analysis, which suggested specific width scalings for initialization and training hyperparameters, and they applied these suggestions to train Vision and Language Transformers in practical setups.
We perform an effective-theory analysis of forward-backward signal propagation in wide and deep Transformers, i.e., residual neural networks with multi-head self-attention blocks and multilayer perceptron blocks. This analysis suggests particular width scalings of initialization and training hyperparameters for these models. We then take up such suggestions, training Vision and Language Transformers in practical setups.