Improving Graph Attention Networks with Large Margin-based Constraints
This work addresses performance issues in graph neural networks for researchers and practitioners, but it is incremental as it builds directly on existing GAT architectures.
The paper tackled the problems of over-fitting and over-smoothing in Graph Attention Networks (GATs) by introducing margin-based constraints on attention weights and graph structure, resulting in significant improvements over previous state-of-the-art methods on all benchmark datasets.
Graph Attention Networks (GATs) are the state-of-the-art neural architecture for representation learning with graphs. GATs learn attention functions that assign weights to nodes so that different nodes have different influences in the feature aggregation steps. In practice, however, induced attention functions are prone to over-fitting due to the increasing number of parameters and the lack of direct supervision on attention weights. GATs also suffer from over-smoothing at the decision boundary of nodes. Here we propose a framework to address their weaknesses via margin-based constraints on attention during training. We first theoretically demonstrate the over-smoothing behavior of GATs and then develop an approach using constraint on the attention weights according to the class boundary and feature aggregation pattern. Furthermore, to alleviate the over-fitting problem, we propose additional constraints on the graph structure. Extensive experiments and ablation studies on common benchmark datasets demonstrate the effectiveness of our method, which leads to significant improvements over the previous state-of-the-art graph attention methods on all datasets.