Graph Transformers Dream of Electric Flow
This work provides foundational insights into how Transformers operate on graph data, potentially benefiting researchers in graph machine learning, though it is an initial step and incremental in nature.
The paper demonstrates that linear Transformers can theoretically and empirically solve graph problems like electric flow and eigenvector decomposition using the graph's incidence matrix, with explicit weight configurations and error bounds validated on synthetic data, and shows improved positional encoding on a molecular regression task.
We show theoretically and empirically that the linear Transformer, when applied to graph data, can implement algorithms that solve canonical problems such as electric flow and eigenvector decomposition. The Transformer has access to information on the input graph only via the graph's incidence matrix. We present explicit weight configurations for implementing each algorithm, and we bound the constructed Transformers' errors by the errors of the underlying algorithms. Our theoretical findings are corroborated by experiments on synthetic data. Additionally, on a real-world molecular regression task, we observe that the linear Transformer is capable of learning a more effective positional encoding than the default one based on Laplacian eigenvectors. Our work is an initial step towards elucidating the inner-workings of the Transformer for graph data. Code is available at https://github.com/chengxiang/LinearGraphTransformer