Enhancing Graph Neural Networks: A Mutual Learning Approach
This work addresses the challenge of efficient model deployment for GNNs, offering an incremental improvement over existing knowledge distillation methods by enabling lightweight models to learn synergistically without a pre-trained teacher.
The paper tackles the problem of deploying high-performance graph neural networks (GNNs) in resource-constrained settings by proposing a collaborative learning framework where ensembles of student GNNs mutually teach each other, achieving improved performance on node and graph classification tasks across multiple datasets.
Knowledge distillation (KD) techniques have emerged as a powerful tool for transferring expertise from complex teacher models to lightweight student models, particularly beneficial for deploying high-performance models in resource-constrained devices. This approach has been successfully applied to graph neural networks (GNNs), harnessing their expressive capabilities to generate node embeddings that capture structural and feature-related information. In this study, we depart from the conventional KD approach by exploring the potential of collaborative learning among GNNs. In the absence of a pre-trained teacher model, we show that relatively simple and shallow GNN architectures can synergetically learn efficient models capable of performing better during inference, particularly in tackling multiple tasks. We propose a collaborative learning framework where ensembles of student GNNs mutually teach each other throughout the training process. We introduce an adaptive logit weighting unit to facilitate efficient knowledge exchange among models and an entropy enhancement technique to improve mutual learning. These components dynamically empower the models to adapt their learning strategies during training, optimizing their performance for downstream tasks. Extensive experiments conducted on three datasets each for node and graph classification demonstrate the effectiveness of our approach.