Transformers are Graph Neural Networks
This work provides a theoretical bridge between NLP and graph learning, offering insights for researchers in both fields, though it is incremental in nature as it reinterprets existing architectures rather than introducing new methods.
The paper establishes a mathematical connection between Transformers and Graph Neural Networks (GNNs), showing that Transformers can be viewed as message passing GNNs on fully connected graphs, with self-attention capturing token relationships and positional encodings providing structural hints, while noting that Transformers are more efficient on modern hardware due to dense matrix operations.
We establish connections between the Transformer architecture, originally introduced for natural language processing, and Graph Neural Networks (GNNs) for representation learning on graphs. We show how Transformers can be viewed as message passing GNNs operating on fully connected graphs of tokens, where the self-attention mechanism capture the relative importance of all tokens w.r.t. each-other, and positional encodings provide hints about sequential ordering or structure. Thus, Transformers are expressive set processing networks that learn relationships among input elements without being constrained by apriori graphs. Despite this mathematical connection to GNNs, Transformers are implemented via dense matrix operations that are significantly more efficient on modern hardware than sparse message passing. This leads to the perspective that Transformers are GNNs currently winning the hardware lottery.