Flexible Infinite-Width Graph Convolutional Neural Networks
This work addresses the theoretical gap in neural network Gaussian processes for graph learning, providing insights into when representation learning matters, though it is incremental in extending existing infinite-width methods.
The paper tackled the problem of understanding whether representation learning is necessary in graph tasks by developing a flexible infinite-width graph convolutional deep kernel machine, finding that representation learning gives noticeable performance improvements for heterophilous node classification tasks but less so for homophilous ones.
A common theoretical approach to understanding neural networks is to take an infinite-width limit, at which point the outputs become Gaussian process (GP) distributed. This is known as a neural network Gaussian process (NNGP). However, the NNGP kernel is fixed and tunable only through a small number of hyperparameters, thus eliminating the possibility of representation learning. This contrasts with finite-width NNs, which are often believed to perform well because they are able to flexibly learn representations for the task at hand. Thus, in simplifying NNs to make them theoretically tractable, NNGPs may eliminate precisely what makes them work well (representation learning). This motivated us to understand whether representation learning is necessary in a range of graph tasks. We develop a precise tool for this task, the graph convolutional deep kernel machine. This is very similar to an NNGP, in that it is an infinite width limit and uses kernels, but comes with a ``knob'' to control the amount of flexibility and hence representation learning. We found that representation learning gives noticeable performance improvements for heterophilous node classification tasks, but less so for homophilous node classification tasks.