The Depth Delusion: Why Transformers Should Be Wider, Not Deeper
This provides insights for scaling language models efficiently, addressing a key architectural design problem for AI researchers and practitioners, though it is incremental in refining existing scaling laws.
The paper tackles the problem of determining optimal depth-width tradeoffs in transformer architectures, finding that width should grow 2.8 times faster than depth and identifying a critical depth beyond which adding layers increases loss despite adding parameters, with empirical validation showing a 64-layer model underperforming a 32-layer model by 0.12 nats at the 7B parameter scale.
Neural scaling laws describe how language model loss decreases with parameters and data, but treat architecture as interchangeable--a billion parameters could arise from a shallow-wide model (10 layers & 8,192 hidden dimension) or a deep-narrow one (80 layers & 2,048 hidden dimension). We propose architecture-conditioned scaling laws decomposing this dependence, finding that optimal depth scales as D* ~ C^0.12 while optimal width scales as W* ~ C^0.34, meaning width should grow 2.8x faster than depth. We discover a critical depth phenomenon: beyond D_crit ~ W^0.44 (sublinear in W), adding layers increases loss despite adding parameters--the Depth Delusion. Empirically, we validate these findings across 30 transformer architectures spanning 17M to 7B parameters, each trained on representative high-compute samples, achieving R^2 = 0.922. Our central finding: at 7B scale, a 64-layer model (6.38B params) underperforms a 32-layer model (6.86B params) by 0.12 nats, despite being significantly deeper. This demonstrates that optimal depth-width tradeoffs persist at the production scale.