Predicting Generalization in Deep Learning via Local Measures of Distortion
This work addresses the problem of predicting generalization performance in deep learning for researchers and practitioners, offering a potentially less expensive alternative to existing methods.
This paper explores the use of local distortion measures, derived from vector quantization methods like PCA, GMMs, and SVMs applied layer-wise to deep features, to predict generalization in deep learning. These measures are shown to correlate well with generalization performance, as demonstrated in the 2020 NeurIPS PGDL challenge.
We study generalization in deep learning by appealing to complexity measures originally developed in approximation and information theory. While these concepts are challenged by the high-dimensional and data-defined nature of deep learning, we show that simple vector quantization approaches such as PCA, GMMs, and SVMs capture their spirit when applied layer-wise to deep extracted features giving rise to relatively inexpensive complexity measures that correlate well with generalization performance. We discuss our results in 2020 NeurIPS PGDL challenge.