HyperInvariances: Amortizing Invariance Learning
This addresses the challenge of expensive and data-intensive invariance learning for neural architectures, offering a more efficient approach for machine learning practitioners.
The paper tackles the problem of unknown ideal invariances in learning tasks by introducing amortized invariance learning, where a hyper-network learns a manifold of feature extractors upfront, enabling rapid invariance adaptation for downstream tasks with comparable or better test performance than conventional methods.
Providing invariances in a given learning task conveys a key inductive bias that can lead to sample-efficient learning and good generalisation, if correctly specified. However, the ideal invariances for many problems of interest are often not known, which has led both to a body of engineering lore as well as attempts to provide frameworks for invariance learning. However, invariance learning is expensive and data intensive for popular neural architectures. We introduce the notion of amortizing invariance learning. In an up-front learning phase, we learn a low-dimensional manifold of feature extractors spanning invariance to different transformations using a hyper-network. Then, for any problem of interest, both model and invariance learning are rapid and efficient by fitting a low-dimensional invariance descriptor an output head. Empirically, this framework can identify appropriate invariances in different downstream tasks and lead to comparable or better test performance than conventional approaches. Our HyperInvariance framework is also theoretically appealing as it enables generalisation-bounds that provide an interesting new operating point in the trade-off between model fit and complexity.