An Essay on Optimization Mystery of Deep Learning
arXiv:1905.07187v13 citations
Originality Synthesis-oriented
AI Analysis
It synthesizes existing research on a theoretical problem for the deep learning community, but is incremental as it reviews prior work.
The paper addresses the optimization mystery in deep learning by reviewing and connecting selected works, but it does not present new results or concrete numbers.
Despite the huge empirical success of deep learning, theoretical understanding of neural networks learning process is still lacking. This is the reason, why some of its features seem "mysterious". We emphasize two mysteries of deep learning: generalization mystery, and optimization mystery. In this essay we review and draw connections between several selected works concerning the latter.