GNNMerge: Merging of GNN Models Without Accessing Training Data
This addresses the challenge of integrating multiple trained GNN models efficiently for practitioners, though it is incremental as it adapts model merging to a new domain.
The paper tackled the problem of merging Graph Neural Network (GNN) models without accessing training data, proposing GNNMerge, which achieved up to 24% higher accuracy than existing methods and over 100 times faster speed compared to training from scratch.
Model merging has gained prominence in machine learning as a method to integrate multiple trained models into a single model without accessing the original training data. While existing approaches have demonstrated success in domains such as computer vision and NLP, their application to Graph Neural Networks (GNNs) remains unexplored. These methods often rely on the assumption of shared initialization, which is seldom applicable to GNNs. In this work, we undertake the first benchmarking study of model merging algorithms for GNNs, revealing their limited effectiveness in this context. To address these challenges, we propose GNNMerge, which utilizes a task-agnostic node embedding alignment strategy to merge GNNs. Furthermore, we establish that under a mild relaxation, the proposed optimization objective admits direct analytical solutions for widely used GNN architectures, significantly enhancing its computational efficiency. Empirical evaluations across diverse datasets, tasks, and architectures establish GNNMerge to be up to 24% more accurate than existing methods while delivering over 2 orders of magnitude speed-up compared to training from scratch.