Hierarchical Learning in Euclidean Neural Networks
This work addresses a fundamental question in machine learning for the physical sciences, aiming to inform design principles for equivariant models, but it is incremental as it builds on prior studies without introducing new methods or broad SOTA results.
The study investigated why equivariant models like Euclidean Neural Networks (e3nn) achieve unexpectedly large advances in 3D learning applications, focusing on electron density prediction, and found that non-scalar hidden features exhibit a hierarchical structure by angular momentum (l) during training, reminiscent of a multipole expansion.
Equivariant machine learning methods have shown wide success at 3D learning applications in recent years. These models explicitly build in the reflection, translation and rotation symmetries of Euclidean space and have facilitated large advances in accuracy and data efficiency for a range of applications in the physical sciences. An outstanding question for equivariant models is why they achieve such larger-than-expected advances in these applications. To probe this question, we examine the role of higher order (non-scalar) features in Euclidean Neural Networks (\texttt{e3nn}). We focus on the previously studied application of \texttt{e3nn} to the problem of electron density prediction, which allows for a variety of non-scalar outputs, and examine whether the nature of the output (scalar $l=0$, vector $l=1$, or higher order $l>1$) is relevant to the effectiveness of non-scalar hidden features in the network. Further, we examine the behavior of non-scalar features throughout training, finding a natural hierarchy of features by $l$, reminiscent of a multipole expansion. We aim for our work to ultimately inform design principles and choices of domain applications for {\tt e3nn} networks.