Learning Graph Quantized Tokenizers
This addresses a bottleneck in geometric deep learning by enabling more efficient and effective graph Transformers, though it is incremental as it builds on existing tokenizer and Transformer methods.
The paper tackles the lack of specialized tokenizers for graph Transformers by introducing GQT, a graph quantized tokenizer that decouples tokenizer training and uses residual vector quantization, achieving state-of-the-art performance on 20 out of 22 benchmarks.
Transformers serve as the backbone architectures of Foundational Models, where domain-specific tokenizers allow them to adapt to various domains. Graph Transformers (GTs) have recently emerged as leading models in geometric deep learning, outperforming Graph Neural Networks (GNNs) in various graph learning tasks. However, the development of tokenizers for graphs has lagged behind other modalities. To address this, we introduce GQT (\textbf{G}raph \textbf{Q}uantized \textbf{T}okenizer), which decouples tokenizer training from Transformer training by leveraging multi-task graph self-supervised learning, yielding robust and generalizable graph tokens. Furthermore, the GQT utilizes Residual Vector Quantization (RVQ) to learn hierarchical discrete tokens, resulting in significantly reduced memory requirements and improved generalization capabilities. By combining the GQT with token modulation, a Transformer encoder achieves state-of-the-art performance on 20 out of 22 benchmarks, including large-scale homophilic and heterophilic datasets.