Functional Attention: From Pairwise Affinities to Functional Correspondences
This work provides a more robust and generalizable approach to operator learning for machine learning applications that deal with continuous fields, offering a resolution-invariant representation.
This paper addresses the challenge of learning mappings between infinite-dimensional function spaces, or operator learning, by introducing Functional Attention. This method reinterprets attention as a functional correspondence between adaptive bases, replacing softmax affinities with structured linear operators, and achieves state-of-the-art performance in tasks like solving PDEs, 3D segmentation, and regression.
Learning mappings between infinite-dimensional function spaces, or operator learning, is essential for many machine learning applications. Although transformer-based operators are popular, they often rely on token-wise attention. These methods treat continuous fields as discrete tokens and usually ignore the global functional structure. We introduce \emph{Functional Attention}, which reinterprets attention as a functional correspondence between adaptive bases. Inspired by geometric functional maps, our method replaces softmax affinities with structured linear operators. This yields a compact, generalizable, resolution-invariant representation that explicitly captures global dependencies. Experiments demonstrate that \emph{Functional Attention} can match state-of-the-art performance in many operator learning tasks, including solving PDEs, 3D segmentation, and regression, while remaining robust to varying discretizations. Project page is available at https://github.com/xjffff/FUNCATTN.