Approximate equivariance via projection-based regularisation
This work addresses the need for efficient and effective approximately equivariant models in machine learning, particularly for applications with imperfect symmetries, representing an incremental improvement over existing approaches.
The paper tackled the problem of balancing equivariance and flexibility in neural networks by introducing a projection-based regularizer that penalizes non-equivariance at an operator level, resulting in consistent outperformance of prior methods in both performance and efficiency with substantial runtime gains.
Equivariance is a powerful inductive bias in neural networks, improving generalisation and physical consistency. Recently, however, non-equivariant models have regained attention, due to their better runtime performance and imperfect symmetries that might arise in real-world applications. This has motivated the development of approximately equivariant models that strike a middle ground between respecting symmetries and fitting the data distribution. Existing approaches in this field usually apply sample-based regularisers which depend on data augmentation at training time, incurring a high sample complexity, in particular for continuous groups such as $SO(3)$. This work instead approaches approximate equivariance via a projection-based regulariser which leverages the orthogonal decomposition of linear layers into equivariant and non-equivariant components. In contrast to existing methods, this penalises non-equivariance at an operator level across the full group orbit, rather than point-wise. We present a mathematical framework for computing the non-equivariance penalty exactly and efficiently in both the spatial and spectral domain. In our experiments, our method consistently outperforms prior approximate equivariance approaches in both model performance and efficiency, achieving substantial runtime gains over sample-based regularisers.