Accelerating Molecular Graph Neural Networks via Knowledge Distillation
This work addresses the computational bottleneck of large-scale molecular GNNs for applications like molecular property prediction and simulations, though it is incremental as it applies existing knowledge distillation techniques to this domain.
The paper tackles the problem of accelerating molecular graph neural networks (GNNs) by using knowledge distillation to reduce computational costs while maintaining accuracy, achieving up to 96.7% and 62.5% gap closure in predictive accuracy for energy and force prediction respectively.
Recent advances in graph neural networks (GNNs) have enabled more comprehensive modeling of molecules and molecular systems, thereby enhancing the precision of molecular property prediction and molecular simulations. Nonetheless, as the field has been progressing to bigger and more complex architectures, state-of-the-art GNNs have become largely prohibitive for many large-scale applications. In this paper, we explore the utility of knowledge distillation (KD) for accelerating molecular GNNs. To this end, we devise KD strategies that facilitate the distillation of hidden representations in directional and equivariant GNNs, and evaluate their performance on the regression task of energy and force prediction. We validate our protocols across different teacher-student configurations and datasets, and demonstrate that they can consistently boost the predictive accuracy of student models without any modifications to their architecture. Moreover, we conduct comprehensive optimization of various components of our framework, and investigate the potential of data augmentation to further enhance performance. All in all, we manage to close the gap in predictive accuracy between teacher and student models by as much as 96.7% and 62.5% for energy and force prediction respectively, while fully preserving the inference throughput of the more lightweight models.