GATE: How to Keep Out Intrusive Neighbors
This addresses a bottleneck in graph neural networks for tasks with heterophilic data, though it is an incremental improvement over existing GATs.
The paper tackles the problem of Graph Attention Networks (GATs) being unable to switch off task-irrelevant neighborhood aggregation, leading to over-smoothing. It proposes GATE, an extension that outperforms GATs on real-world heterophilic datasets by down-weighting unrelated neighbors.
Graph Attention Networks (GATs) are designed to provide flexible neighborhood aggregation that assigns weights to neighbors according to their importance. In practice, however, GATs are often unable to switch off task-irrelevant neighborhood aggregation, as we show experimentally and analytically. To address this challenge, we propose GATE, a GAT extension that holds three major advantages: i) It alleviates over-smoothing by addressing its root cause of unnecessary neighborhood aggregation. ii) Similarly to perceptrons, it benefits from higher depth as it can still utilize additional layers for (non-)linear feature transformations in case of (nearly) switched-off neighborhood aggregation. iii) By down-weighting connections to unrelated neighbors, it often outperforms GATs on real-world heterophilic datasets. To further validate our claims, we construct a synthetic test bed to analyze a model's ability to utilize the appropriate amount of neighborhood aggregation, which could be of independent interest.