Last Layer Marginal Likelihood for Invariance Learning
This work addresses the challenge of automating invariance learning for machine learning practitioners, but it is incremental as it builds on existing approximations and shows limited applicability.
The paper tackled the problem of learning invariances in neural networks using the marginal likelihood, specifically by approximating it for the last layer only, and achieved partial success on standard benchmarks and a medical imaging dataset in low-data regimes, but failed on CIFAR10 due to network complexity.
Data augmentation is often used to incorporate inductive biases into models. Traditionally, these are hand-crafted and tuned with cross validation. The Bayesian paradigm for model selection provides a path towards end-to-end learning of invariances using only the training data, by optimising the marginal likelihood. Computing the marginal likelihood is hard for neural networks, but success with tractable approaches that compute the marginal likelihood for the last layer only raises the question of whether this convenient approach might be employed for learning invariances. We show partial success on standard benchmarks, in the low-data regime and on a medical imaging dataset by designing a custom optimisation routine. Introducing a new lower bound to the marginal likelihood allows us to perform inference for a larger class of likelihood functions than before. On the other hand, we demonstrate failure modes on the CIFAR10 dataset, where the last layer approximation is not sufficient due to the increased complexity of our neural network. Our results indicate that once more sophisticated approximations become available the marginal likelihood is a promising approach for invariance learning in neural networks.