Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?
This addresses the problem of optimizing model selection for scaling in AI research, providing insights that could impact how architectures are evaluated, though it is incremental as it builds on existing scaling studies.
The paper investigates how different model architectures and their inductive biases affect scaling behavior in machine learning, finding that architecture significantly influences performance and the best model can vary at different scales.
There have been a lot of interest in the scaling properties of Transformer models. However, not much has been done on the front of investigating the effect of scaling properties of different inductive biases and model architectures. Do model architectures scale differently? If so, how does inductive bias affect scaling behaviour? How does this influence upstream (pretraining) and downstream (transfer)? This paper conducts a systematic study of scaling behaviour of ten diverse model architectures such as Transformers, Switch Transformers, Universal Transformers, Dynamic convolutions, Performers, and recently proposed MLP-Mixers. Via extensive experiments, we show that (1) architecture is an indeed an important consideration when performing scaling and (2) the best performing model can fluctuate at different scales. We believe that the findings outlined in this work has significant implications to how model architectures are currently evaluated in the community.