Dynamical structure of vanishing gradient and overfitting in multi-layer perceptrons
For machine learning theorists, this work offers a rigorous dynamical analysis of two fundamental training problems, though it is incremental as it builds on existing models.
This paper provides a dynamical description of vanishing gradient and overfitting in multi-layer perceptrons, showing that learning passes through plateau and near-optimal saddle regions before converging to an overfitting attractor. It proves that MLPs trained on finite noisy datasets necessarily converge to overfitting solutions.
Vanishing gradient and overfitting are two of the most extensively studied problems in the literature about machine learning. However, they are frequently considered in some asymptotic setting, which obscure the underlying dynamical mechanisms responsible for their emergence. In this paper, we aim to provide a clear dynamical description of learning in multi-layer perceptrons. To this end, we introduce a minimal model, inspired by studies by Fukumizu and Amari, to investigate vanishing gradients and overfitting in MLPs trained via gradient descent. Within this model, we show that the learning dynamics may pass through plateau regions and near-optimal regions during training, both of which consist of saddle structures, before ultimately converging to the overfitting region. Under suitable conditions on the training dataset, we prove that, with high probability, the overfitting region collapses to a single attractor modulo symmetry, which corresponds to the overfitting. Moreover, we show that any MLP trained on a finite noisy dataset cannot converge to the theoretical optimum and instead necessarily converges to an overfitting solution.