Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
This addresses a foundational issue in Transformer architecture design, offering insights for improving training efficiency in large language models, though it is incremental as it builds on existing normalization strategies.
The paper tackled the problem of selecting layer normalization strategies to stabilize training and speed convergence in large-scale Transformers, showing that Peri-LN achieves more balanced variance growth, steadier gradient flow, and convergence stability in experiments up to 3.2B parameters.
Selecting a layer normalization (LN) strategy that stabilizes training and speeds convergence in Transformers remains difficult, even for today's large language models (LLM). We present a comprehensive analytical foundation for understanding how different LN strategies influence training dynamics in large-scale Transformers. Until recently, Pre-LN and Post-LN have long dominated practices despite their limitations in large-scale training. However, several open-source models have recently begun silently adopting a third strategy without much explanation. This strategy places normalization layer peripherally around sublayers, a design we term Peri-LN. While Peri-LN has demonstrated promising performance, its precise mechanisms and benefits remain almost unexplored. Our in-depth analysis delineates the distinct behaviors of LN strategies, showing how each placement shapes activation variance and gradient propagation. To validate our theoretical insight, we conduct extensive experiments on Transformers up to $3.2$B parameters, showing that Peri-LN consistently achieves more balanced variance growth, steadier gradient flow, and convergence stability. Our results suggest that Peri-LN warrants broader consideration for large-scale Transformer architectures, providing renewed insights into the optimal placement of LN.