Practical recommendations for gradient-based training of deep architectures
This offers incremental guidance for practitioners in machine learning to improve training efficiency and address common difficulties in deep architectures.
The paper tackles the challenge of hyperparameter tuning in deep learning by providing practical recommendations for gradient-based training, focusing on efficiently training and debugging large-scale deep neural networks.
Learning algorithms related to artificial neural networks and in particular for Deep Learning may seem to involve many bells and whistles, called hyper-parameters. This chapter is meant as a practical guide with recommendations for some of the most commonly used hyper-parameters, in particular in the context of learning algorithms based on back-propagated gradient and gradient-based optimization. It also discusses how to deal with the fact that more interesting results can be obtained when allowing one to adjust many hyper-parameters. Overall, it describes elements of the practice used to successfully and efficiently train and debug large-scale and often deep multi-layer neural networks. It closes with open questions about the training difficulties observed with deeper architectures.