Generalization bounds via distillation
This provides a theoretical foundation for improving generalization in neural networks through distillation, which is incremental as it builds on existing distillation techniques.
The paper tackles the problem of poor generalization bounds in high-complexity neural networks by showing that distilling them into low-complexity networks yields nearly identical predictions with vastly smaller generalization bounds, assuming well-behaved data augmentation, and demonstrates this with experiments on CIFAR and MNIST datasets.
This paper theoretically investigates the following empirical phenomenon: given a high-complexity network with poor generalization bounds, one can distill it into a network with nearly identical predictions but low complexity and vastly smaller generalization bounds. The main contribution is an analysis showing that the original network inherits this good generalization bound from its distillation, assuming the use of well-behaved data augmentation. This bound is presented both in an abstract and in a concrete form, the latter complemented by a reduction technique to handle modern computation graphs featuring convolutional layers, fully-connected layers, and skip connections, to name a few. To round out the story, a (looser) classical uniform convergence analysis of compression is also presented, as well as a variety of experiments on cifar and mnist demonstrating similar generalization performance between the original network and its distillation.