Deep invariant networks with differentiable augmentation layers
This addresses the problem of efficiently enforcing invariances in machine learning models for practitioners, offering a versatile solution applicable beyond computer vision, though it is incremental as it builds on existing augmentation techniques.
The paper tackles the challenge of learning data invariances without prior knowledge or held-out data by introducing learnable augmentation layers integrated into neural networks, achieving comparable results to state-of-the-art automatic data augmentation methods while being easier and faster to train.
Designing learning systems which are invariant to certain data transformations is critical in machine learning. Practitioners can typically enforce a desired invariance on the trained model through the choice of a network architecture, e.g. using convolutions for translations, or using data augmentation. Yet, enforcing true invariance in the network can be difficult, and data invariances are not always known a piori. State-of-the-art methods for learning data augmentation policies require held-out data and are based on bilevel optimization problems, which are complex to solve and often computationally demanding. In this work we investigate new ways of learning invariances only from the training data. Using learnable augmentation layers built directly in the network, we demonstrate that our method is very versatile. It can incorporate any type of differentiable augmentation and be applied to a broad class of learning problems beyond computer vision. We provide empirical evidence showing that our approach is easier and faster to train than modern automatic data augmentation techniques based on bilevel optimization, while achieving comparable results. Experiments show that while the invariances transferred to a model through automatic data augmentation are limited by the model expressivity, the invariance yielded by our approach is insensitive to it by design.