Euclidean, Projective, Conformal: Choosing a Geometric Algebra for Equivariant Transformers
This work addresses the problem of selecting appropriate geometric algebras for equivariant transformers in 3D geometric deep learning, offering a flexible framework with incremental improvements over existing methods.
The authors generalized the Geometric Algebra Transformer (GATr) into a blueprint for scalable transformer architectures using any geometric algebra, evaluating Euclidean, projective, and conformal versions for 3D data. They found that Euclidean models are computationally cheap but less sample-efficient, projective models lack expressiveness, while conformal and improved projective versions are powerful and performant.
The Geometric Algebra Transformer (GATr) is a versatile architecture for geometric deep learning based on projective geometric algebra. We generalize this architecture into a blueprint that allows one to construct a scalable transformer architecture given any geometric (or Clifford) algebra. We study versions of this architecture for Euclidean, projective, and conformal algebras, all of which are suited to represent 3D data, and evaluate them in theory and practice. The simplest Euclidean architecture is computationally cheap, but has a smaller symmetry group and is not as sample-efficient, while the projective model is not sufficiently expressive. Both the conformal algebra and an improved version of the projective algebra define powerful, performant architectures.