On the Benefits of Invariance in Neural Networks
This work addresses the lack of theoretical understanding for invariance methods in deep learning, providing insights for researchers and practitioners on when to prefer specific approaches, though it is incremental in building on existing techniques.
The paper analyzes the theoretical benefits and limitations of data augmentation and feature averaging for incorporating invariance in neural networks, proving that data augmentation improves risk and gradient estimates with a PAC-Bayes bound, while feature averaging reduces generalization error for convex losses and tightens bounds, supported by empirical evidence.
Many real world data analysis problems exhibit invariant structure, and models that take advantage of this structure have shown impressive empirical performance, particularly in deep learning. While the literature contains a variety of methods to incorporate invariance into models, theoretical understanding is poor and there is no way to assess when one method should be preferred over another. In this work, we analyze the benefits and limitations of two widely used approaches in deep learning in the presence of invariance: data augmentation and feature averaging. We prove that training with data augmentation leads to better estimates of risk and gradients thereof, and we provide a PAC-Bayes generalization bound for models trained with data augmentation. We also show that compared to data augmentation, feature averaging reduces generalization error when used with convex losses, and tightens PAC-Bayes bounds. We provide empirical support of these theoretical results, including a demonstration of why generalization may not improve by training with data augmentation: the `learned invariance' fails outside of the training distribution.