Benchmarking GNNs for OOD Materials Property Prediction with Uncertainty Quantification
This work addresses the challenge of reliable model selection under distribution shifts for materials discovery, though it is incremental as it builds on existing methods with new benchmarks and metrics.
The authors tackled the problem of evaluating graph neural networks (GNNs) for out-of-distribution (OOD) materials property prediction with uncertainty quantification, resulting in a benchmark framework (MatUQ) that shows uncertainty-aware training reduces errors by an average of 70.6% and reveals no single model dominates across all tasks.
We present MatUQ, a benchmark framework for evaluating graph neural networks (GNNs) on out-of-distribution (OOD) materials property prediction with uncertainty quantification (UQ). MatUQ comprises 1,375 OOD prediction tasks constructed from six materials datasets using five OFM-based and a newly proposed structure-aware splitting strategy, SOAP-LOCO, which captures local atomic environments more effectively. We evaluate 12 representative GNN models under a unified uncertainty-aware training protocol that combines Monte Carlo Dropout and Deep Evidential Regression (DER), and introduce a novel uncertainty metric, D-EviU, which shows the strongest correlation with prediction errors in most tasks. Our experiments yield two key findings. First, the uncertainty-aware training approach significantly improves model prediction accuracy, reducing errors by an average of 70.6\% across challenging OOD scenarios. Second, the benchmark reveals that no single model dominates universally: earlier models such as SchNet and ALIGNN remain competitive, while newer models like CrystalFramer and SODNet demonstrate superior performance on specific material properties. These results provide practical insights for selecting reliable models under distribution shifts in materials discovery.