Reducing Down(stream)time: Pretraining Molecular GNNs using Heterogeneous AI Accelerators
This work addresses the challenge of implementing transfer learning for chemistry, specifically for molecular GNNs, by demonstrating significant speedups in training and finetuning, though it is incremental as it applies existing methods to a new domain with hardware acceleration.
The authors tackled the problem of slow training times for molecular graph neural networks by pretraining on a large dataset of 2.7 million water clusters using Graphcore IPUs, reducing training time from 2.7 days to 1.2 hours and enabling efficient finetuning for downstream tasks in 8.3 hours and 28 minutes.
The demonstrated success of transfer learning has popularized approaches that involve pretraining models from massive data sources and subsequent finetuning towards a specific task. While such approaches have become the norm in fields such as natural language processing, implementation and evaluation of transfer learning approaches for chemistry are in the early stages. In this work, we demonstrate finetuning for downstream tasks on a graph neural network (GNN) trained over a molecular database containing 2.7 million water clusters. The use of Graphcore IPUs as an AI accelerator for training molecular GNNs reduces training time from a reported 2.7 days on 0.5M clusters to 1.2 hours on 2.7M clusters. Finetuning the pretrained model for downstream tasks of molecular dynamics and transfer to a different potential energy surface took only 8.3 hours and 28 minutes, respectively, on a single GPU.