Multiple Descents in Deep Learning as a Sequence of Order-Chaos Transitions in LSTM Networks
This work provides a theoretical explanation for performance cycles in overfitted LSTMs, offering guidance for selecting optimal training steps, but is incremental as it extends known phase transition concepts to a specific architecture.
The paper discovers a 'multiple-descent' phenomenon in LSTM training, where test performance oscillates in cycles after overtraining, and identifies that optimal performance occurs at the first order-to-chaos phase transition, where the 'edge of chaos' is widest.
We observe a novel `multiple-descent' phenomenon during the learning process of a recurrent neural network called long-short-term memory (LSTM) networks during its training on real-world task, in which the performance goes through long cycles of up and down trends multiple times after the model is overtrained. By carrying out asymptotic stability analysis of the models, we found that the cycles in performance -- indicated by loss function in test data -- are closely associated with the phase transition process between order and chaos of the model, and the local optimal training step are consistently at the critical transition point between the two phases. More importantly, the most optimal point of the model usually occurs at the first transition from order to chaos, where the `width' of the `edge of chaos' is often the widest, allowing the best exploration of weight configurations for learning.