GIST: Gauge-Invariant Spectral Transformers for Scalable Graph Neural Operators
This addresses a critical problem for researchers and practitioners in graph neural networks and neural operators by enabling scalable and invariant learning across different graph discretizations, though it is an incremental improvement over existing transformer methods.
The paper tackled the challenge of adapting transformer positional encoding to meshes and graph-structured data, which suffers from computational inefficiency and gauge invariance issues, by proposing GIST, a graph transformer architecture that achieves O(N) complexity and preserves gauge invariance, resulting in state-of-the-art performance on benchmarks like 99.50% micro-F1 on PPI and scaling to mesh-based datasets with up to 750K nodes.
Adapting transformer positional encoding to meshes and graph-structured data presents significant computational challenges: exact spectral methods require cubic-complexity eigendecomposition and can inadvertently break gauge invariance through numerical solver artifacts, while efficient approximate methods sacrifice gauge symmetry by design. Both failure modes cause catastrophic generalization in inductive learning, where models trained with one set of numerical choices fail when encountering different spectral decompositions of similar graphs or discretizations of the same mesh. We propose GIST (Gauge-Invariant Spectral Transformers), a new graph transformer architecture that resolves this challenge by achieving end-to-end $\mathcal{O}(N)$ complexity through random projections while algorithmically preserving gauge invariance via inner-product-based attention on the projected embeddings. We prove GIST achieves discretization-invariant learning with bounded mismatch error, enabling parameter transfer across arbitrary mesh resolutions for neural operator applications. Empirically, GIST matches state-of-the-art on standard graph benchmarks (e.g., achieving 99.50% micro-F1 on PPI) while uniquely scaling to mesh-based Neural Operator benchmarks with up to 750K nodes, achieving state-of-the-art aerodynamic prediction on the challenging DrivAerNet and DrivAerNet++ datasets.