Superposition unifies power-law training dynamics
This finding may impact a wide range of neural networks, including large language models, by explaining rapid training dynamics.
The paper tackled the role of feature superposition in power-law training dynamics, discovering that superposition induces a universal exponent of ~1, leading to up to tenfold acceleration compared to sequential learning.
We investigate the role of feature superposition in the emergence of power-law training dynamics using a teacher-student framework. We first derive an analytic theory for training without superposition, establishing that the power-law training exponent depends on both the input data statistics and channel importance. Remarkably, we discover that a superposition bottleneck induces a transition to a universal power-law exponent of $\sim 1$, independent of data and channel statistics. This one over time training with superposition represents an up to tenfold acceleration compared to the purely sequential learning that takes place in the absence of superposition. Our finding that superposition leads to rapid training with a data-independent power law exponent may have important implications for a wide range of neural networks that employ superposition, including production-scale large language models.