Space filling positionality and the Spiroformer
This addresses the challenge of applying transformers to non-sequential geometric data, which could benefit fields like computational geometry or physics, but it appears incremental as it adapts existing methods to new domains.
The paper tackles the problem of generalizing transformer models to geometric domains like manifolds, which lack a global order, by proposing attention heads that follow a space-filling curve, and demonstrates this with the Spiroformer on the 2-sphere, achieving results that show improved handling of sequential data in such domains.
Transformers excel when dealing with sequential data. Generalizing transformer models to geometric domains, such as manifolds, we encounter the problem of not having a well-defined global order. We propose a solution with attention heads following a space-filling curve. As a first experimental example, we present the Spiroformer, a transformer that follows a polar spiral on the $2$-sphere.