The Depth-to-Width Interplay in Self-Attention
This addresses the problem of inefficient scaling in large language models for researchers and engineers, offering specific guidelines to optimize network architecture, though it is incremental as it builds on prior observations of depth-inefficiency.
The paper tackles the depth-inefficiency in self-attention networks by theoretically predicting and empirically validating a width-dependent transition between depth-efficiency and depth-inefficiency, providing quantitative guidelines for optimal depth-to-width allocation, such as marking a width of 30K as optimal for a 1-trillion parameter network.
Self-attention architectures, which are rapidly pushing the frontier in natural language processing, demonstrate a surprising depth-inefficient behavior: previous works indicate that increasing the internal representation (network width) is just as useful as increasing the number of self-attention layers (network depth). We theoretically predict a width-dependent transition between depth-efficiency and depth-inefficiency in self-attention. We conduct systematic empirical ablations on networks of depths 6 to 48 that clearly reveal the theoretically predicted behaviors, and provide explicit quantitative suggestions regarding the optimal depth-to-width allocation for a given self-attention network size. The race towards beyond 1-Trillion parameter language models renders informed guidelines for increasing self-attention depth and width in tandem an essential ingredient. Our guidelines elucidate the depth-to-width trade-off in self-attention networks of sizes up to the scale of GPT3 (which we project to be too deep for its size), and beyond, marking an unprecedented width of 30K as optimal for a 1-Trillion parameter network.