Training Diverse Graph Experts for Ensembles: A Systematic Empirical Study
This work addresses the need for effective Mixture-of-Experts frameworks in graph learning, providing incremental insights through a comprehensive evaluation of diversification strategies.
The paper tackles the problem of limited performance in single Graph Neural Networks (GNNs) due to graph heterogeneity by conducting a systematic empirical study of diversification techniques for GNN ensembles, resulting in actionable guidance for training diverse experts and improving ensemble performance across 14 benchmarks.
Graph Neural Networks (GNNs) have become essential tools for learning on relational data, yet the performance of a single GNN is often limited by the heterogeneity present in real-world graphs. Recent advances in Mixture-of-Experts (MoE) frameworks demonstrate that assembling multiple, explicitly diverse GNNs with distinct generalization patterns can significantly improve performance. In this work, we present the first systematic empirical study of expert-level diversification techniques for GNN ensembles. Evaluating 20 diversification strategies -- including random re-initialization, hyperparameter tuning, architectural variation, directionality modeling, and training data partitioning -- across 14 node classification benchmarks, we construct and analyze over 200 ensemble variants. Our comprehensive evaluation examines each technique in terms of expert diversity, complementarity, and ensemble performance. We also uncovers mechanistic insights into training maximally diverse experts. These findings provide actionable guidance for expert training and the design of effective MoE frameworks on graph data. Our code is available at https://github.com/Hydrapse/bench-gnn-diversification.