Learning Invariant Weights in Neural Networks
This addresses the challenge of incorporating symmetries into neural networks for improved predictive power, but it appears incremental as it adapts an existing Gaussian Process approach to neural networks.
The authors tackled the problem of learning invariances from data in neural networks, proposing a method that minimizes a lower bound on the marginal likelihood to achieve higher performance, though no concrete numbers are provided.
Assumptions about invariances or symmetries in data can significantly increase the predictive power of statistical models. Many commonly used models in machine learning are constraint to respect certain symmetries in the data, such as translation equivariance in convolutional neural networks, and incorporation of new symmetry types is actively being studied. Yet, efforts to learn such invariances from the data itself remains an open research problem. It has been shown that marginal likelihood offers a principled way to learn invariances in Gaussian Processes. We propose a weight-space equivalent to this approach, by minimizing a lower bound on the marginal likelihood to learn invariances in neural networks resulting in naturally higher performing models.