Spectral Alignment as Predictor of Loss Explosion in Neural Network Training
This addresses the issue of safeguarding expensive training runs for deep learning practitioners, offering a practical tool with low computational overhead.
The paper tackled the problem of predicting loss explosions in neural network training by introducing Spectral Alignment (SA), a metric that monitors alignment between layer inputs and weight matrix singular vectors, and demonstrated that SA provides earlier and clearer warnings of training divergence than traditional metrics.
Loss explosions in training deep neural networks can nullify multi-million dollar training runs. Conventional monitoring metrics like weight and gradient norms are often lagging and ambiguous predictors, as their values vary dramatically across different models and even between layers of the same model, making it difficult to establish a unified standard for detecting impending failure. We introduce Spectral Alignment (SA), a novel, theoretically-grounded metric that monitors the distributional alignment between layer inputs and the principal singular vectors of weight matrices. We show that a collapse in the sign diversity of this alignment is a powerful early predictor of representational collapse and training divergence. Empirical results on language models demonstrate that monitoring the SA distribution provides a significantly earlier and clearer warning of loss explosions than traditional scalar metrics. SA's low computational overhead makes it a practical tool for safeguarding model training.