Predicting the success of Gradient Descent for a particular Dataset-Architecture-Initialization (DAI)
This addresses the challenge of inefficient training in deep learning by enabling early stopping for poorly performing configurations, though it is incremental as it builds on existing methods for training analysis.
The paper tackles the problem of predicting whether gradient descent will successfully train a given dataset-architecture-initialization combination, showing that analyzing singular values of hidden layers can predict success early in training, with experiments indicating this method is more accurate than using early validation accuracy.
Despite their massive success, training successful deep neural networks still largely relies on experimentally choosing an architecture, hyper-parameters, initialization, and training mechanism. In this work, we focus on determining the success of standard gradient descent method for training deep neural networks on a specified dataset, architecture, and initialization (DAI) combination. Through extensive systematic experiments, we show that the evolution of singular values of the matrix obtained from the hidden layers of a DNN can aid in determining the success of gradient descent technique to train a DAI, even in the absence of validation labels in the supervised learning paradigm. This phenomenon can facilitate early give-up, stopping the training of neural networks which are predicted to not generalize well, early in the training process. Our experimentation across multiple datasets, architectures, and initializations reveals that the proposed scores can more accurately predict the success of a DAI than simply relying on the validation accuracy at earlier epochs to make a judgment.