Optimization for deep learning: theory and algorithms
It synthesizes existing knowledge on optimization challenges in deep learning, making it incremental for researchers in the field.
The paper provides an overview of optimization algorithms and theory for training neural networks, addressing issues like gradient explosion/vanishing and reviewing methods such as SGD and adaptive gradient techniques, but does not present new experimental results or concrete numerical improvements.
When and why can a neural network be successfully trained? This article provides an overview of optimization algorithms and theory for training neural networks. First, we discuss the issue of gradient explosion/vanishing and the more general issue of undesirable spectrum, and then discuss practical solutions including careful initialization and normalization methods. Second, we review generic optimization methods used in training neural networks, such as SGD, adaptive gradient methods and distributed methods, and theoretical results for these algorithms. Third, we review existing research on the global issues of neural network training, including results on bad local minima, mode connectivity, lottery ticket hypothesis and infinite-width analysis.