Scaling Graph Transformers: A Comparative Study of Sparse and Dense Attention
This work provides guidance for researchers and practitioners in machine learning on selecting attention types for graph transformers, but it is incremental as it builds on existing methods without introducing new paradigms.
The paper compares dense and sparse attention mechanisms in graph transformers to address their trade-offs and identify appropriate use cases, analyzing their performance without providing specific numerical results.
Graphs have become a central representation in machine learning for capturing relational and structured data across various domains. Traditional graph neural networks often struggle to capture long-range dependencies between nodes due to their local structure. Graph transformers overcome this by using attention mechanisms that allow nodes to exchange information globally. However, there are two types of attention in graph transformers: dense and sparse. In this paper, we compare these two attention mechanisms, analyze their trade-offs, and highlight when to use each. We also outline current challenges and problems in designing attention for graph transformers.