Stable Language Model Pre-training by Reducing Embedding Variability
This addresses stability issues in pre-training for language model developers, offering an incremental improvement with practical efficiency gains.
The paper tackles the problem of unstable pre-training in language models by proposing Token Embedding Variability as an efficient proxy for assessing stability and introducing Multi-head Low-Rank Attention to prevent gradient explosion, resulting in increased stability and lower perplexity, especially in deeper models.
Stable pre-training is essential for achieving better-performing language models. However, tracking pre-training stability by calculating gradient variance at every step is impractical due to the significant computational costs. We explore Token Embedding Variability (TEV) as a simple and efficient proxy for assessing pre-training stability in language models with pre-layer normalization, given that shallower layers are more prone to gradient explosion (section 2.2). Moreover, we propose Multi-head Low-Rank Attention (MLRA) as an architecture to alleviate such instability by limiting the exponential growth of output embedding variance, thereby preventing the gradient explosion (section 3.2). Empirical results on GPT-2 with MLRA demonstrate increased stability and lower perplexity, particularly in deeper models.