Representation Based Complexity Measures for Predicting Generalization in Deep Learning
This work provides practical tools for researchers and practitioners to understand and predict generalization in deep learning models, which is a critical challenge in the field.
This paper explores generalization in deep neural networks by introducing practical complexity measures derived from the quality of internal representations, inspired by human visual system theories. These measures are designed to predict generalization behavior in deep models and formed the basis of a winning solution for the NeurIPS 2020 Predicting Generalization in Deep Learning competition.
Deep Neural Networks can generalize despite being significantly overparametrized. Recent research has tried to examine this phenomenon from various view points and to provide bounds on the generalization error or measures predictive of the generalization gap based on these viewpoints, such as norm-based, PAC-Bayes based, and margin-based analysis. In this work, we provide an interpretation of generalization from the perspective of quality of internal representations of deep neural networks, based on neuroscientific theories of how the human visual system creates invariant and untangled object representations. Instead of providing theoretical bounds, we demonstrate practical complexity measures which can be computed ad-hoc to uncover generalization behaviour in deep models. We also provide a detailed description of our solution that won the NeurIPS competition on Predicting Generalization in Deep Learning held at NeurIPS 2020. An implementation of our solution is available at https://github.com/parthnatekar/pgdl.