A tradeoff between universality of equivariant models and learnability of symmetries
This work addresses a fundamental tradeoff in machine learning for researchers developing equivariant models, but it is incremental as it builds on existing theorems and analyses.
The authors prove an impossibility result showing that under certain conditions, one cannot simultaneously learn symmetries and equivariant functions using an equivariant ansatz, and they analyze neural networks like linearly equivariant and group-convolutional types to generalize theoretical results to non-homogeneous spaces.
We prove an impossibility result, which in the context of function learning says the following: under certain conditions, it is impossible to simultaneously learn symmetries and functions equivariant under them using an ansatz consisting of equivariant functions. To formalize this statement, we carefully study notions of approximation for groups and semigroups. We analyze certain families of neural networks for whether they satisfy the conditions of the impossibility result: what we call ``linearly equivariant'' networks, and group-convolutional networks. A lot can be said precisely about linearly equivariant networks, making them theoretically useful. On the practical side, our analysis of group-convolutional neural networks allows us generalize the well-known ``convolution is all you need'' theorem to non-homogeneous spaces. We additionally find an important difference between group convolution and semigroup convolution.