Universal One-third Time Scaling in Learning Peaked Distributions
This addresses the problem of computationally expensive LLM training by explaining a key bottleneck, potentially enabling more efficient training methods.
The paper identifies that using softmax and cross-entropy when learning peaked probability distributions (like next-token distributions in LLMs) creates a fundamental optimization bottleneck, leading to power-law loss convergence with a universal exponent of 1/3. This provides a mechanistic explanation for observed neural scaling laws in LLM training.
Training large language models (LLMs) is computationally expensive, partly because the loss exhibits slow power-law convergence whose origin remains debatable. Through systematic analysis of toy models and empirical evaluation of LLMs, we show that this behavior can arise intrinsically from the use of softmax and cross-entropy. When learning peaked probability distributions, e.g., next-token distributions, these components yield power-law vanishing losses and gradients, creating a fundamental optimization bottleneck. This ultimately leads to power-law time scaling of the loss with a universal exponent of $1/3$. Our results provide a mechanistic explanation for observed neural scaling and suggest new directions for improving LLM training efficiency.