SGHormer: An Energy-Saving Graph Transformer Driven by Spikes
This addresses the scalability and energy efficiency problem for large-scale graph tasks, representing an incremental improvement by applying biologically inspired spiking methods to an existing framework.
The paper tackles the high energy consumption and computational overhead of Graph Transformers by proposing SGHormer, a spiking-based graph transformer that converts full-precision embeddings into sparse, binarized spikes, achieving comparable performance to full-precision models with extremely low energy consumption.
Graph Transformers (GTs) with powerful representation learning ability make a huge success in wide range of graph tasks. However, the costs behind outstanding performances of GTs are higher energy consumption and computational overhead. The complex structure and quadratic complexity during attention calculation in vanilla transformer seriously hinder its scalability on the large-scale graph data. Though existing methods have made strides in simplifying combinations among blocks or attention-learning paradigm to improve GTs' efficiency, a series of energy-saving solutions originated from biologically plausible structures are rarely taken into consideration when constructing GT framework. To this end, we propose a new spiking-based graph transformer (SGHormer). It turns full-precision embeddings into sparse and binarized spikes to reduce memory and computational costs. The spiking graph self-attention and spiking rectify blocks in SGHormer explicitly capture global structure information and recover the expressive power of spiking embeddings, respectively. In experiments, SGHormer achieves comparable performances to other full-precision GTs with extremely low computational energy consumption. The results show that SGHomer makes a remarkable progress in the field of low-energy GTs.