Learning Invariances in Neural Networks
This addresses the challenge of designing invariant models for practitioners in machine learning, though it is incremental as it builds on existing augmentation techniques.
The paper tackles the problem of automatically learning which invariances and equivariances are beneficial for neural networks, rather than relying on prior knowledge, by parameterizing augmentations and optimizing them alongside model parameters. The result is a method that successfully recovers appropriate invariances across tasks like image classification and molecular prediction, as demonstrated on training data alone.
Invariances to translations have imbued convolutional neural networks with powerful generalization properties. However, we often do not know a priori what invariances are present in the data, or to what extent a model should be invariant to a given symmetry group. We show how to \emph{learn} invariances and equivariances by parameterizing a distribution over augmentations and optimizing the training loss simultaneously with respect to the network parameters and augmentation parameters. With this simple procedure we can recover the correct set and extent of invariances on image classification, regression, segmentation, and molecular property prediction from a large space of augmentations, on training data alone.