Exploring Generalization in Deep Learning
This work addresses the fundamental problem of understanding generalization in deep learning for researchers, but it is incremental as it builds on existing theories without introducing new methods.
The paper investigates factors driving generalization in deep networks, examining norm-based control, sharpness, and robustness, and connects sharpness to PAC-Bayes theory to analyze their explanatory power for observed phenomena.
With a goal of understanding what drives generalization in deep networks, we consider several recently suggested explanations, including norm-based control, sharpness and robustness. We study how these measures can ensure generalization, highlighting the importance of scale normalization, and making a connection between sharpness and PAC-Bayes theory. We then investigate how well the measures explain different observed phenomena.