Generalization by design: Shortcuts to Generalization in Deep Learning
This work addresses the fundamental challenge of generalization in deep learning for researchers and practitioners, offering a novel perspective that could lead to better-architected models, though it is incremental in building on existing regularization techniques.
The paper tackles the problem of understanding and improving generalization in deep learning by proposing that good generalization can be encoded into network structure through bounded spectral products, leading to a geometric regularizer. It demonstrates that this approach enables extreme accuracy and generalization in deep models, with practical verification on datasets like MNIST and CIFAR10.
We take a geometrical viewpoint and present a unifying view on supervised deep learning with the Bregman divergence loss function - this entails frequent classification and prediction tasks. Motivated by simulations we suggest that there is principally no implicit bias of vanilla stochastic gradient descent training of deep models towards "simpler" functions. Instead, we show that good generalization may be instigated by bounded spectral products over layers leading to a novel geometric regularizer. It is revealed that in deep enough models such a regularizer enables both, extreme accuracy and generalization, to be reached. We associate popular regularization techniques like weight decay, drop out, batch normalization, and early stopping with this perspective. Backed up by theory we further demonstrate that "generalization by design" is practically possible and that good generalization may be encoded into the structure of the network. We design two such easy-to-use structural regularizers that insert an additional \textit{generalization layer} into a model architecture, one with a skip connection and another one with drop-out. We verify our theoretical results in experiments on various feedforward and convolutional architectures, including ResNets, and datasets (MNIST, CIFAR10, synthetic data). We believe this work opens up new avenues of research towards better generalizing architectures.