A Fair Comparison of Graph Neural Networks for Graph Classification
This work provides a rigorous evaluation framework to improve reproducibility and fairness in comparing GNNs for graph classification, which is incremental but essential for the field's development.
The paper addresses the lack of reproducibility in graph neural network (GNN) research by conducting over 47,000 experiments in a controlled framework to re-evaluate five popular models across nine benchmarks, revealing that structural information is underutilized in some datasets.
Experimental reproducibility and replicability are critical topics in machine learning. Authors have often raised concerns about their lack in scientific publications to improve the quality of the field. Recently, the graph representation learning field has attracted the attention of a wide research community, which resulted in a large stream of works. As such, several Graph Neural Network models have been developed to effectively tackle graph classification. However, experimental procedures often lack rigorousness and are hardly reproducible. Motivated by this, we provide an overview of common practices that should be avoided to fairly compare with the state of the art. To counter this troubling trend, we ran more than 47000 experiments in a controlled and uniform framework to re-evaluate five popular models across nine common benchmarks. Moreover, by comparing GNNs with structure-agnostic baselines we provide convincing evidence that, on some datasets, structural information has not been exploited yet. We believe that this work can contribute to the development of the graph learning field, by providing a much needed grounding for rigorous evaluations of graph classification models.