LaMM: Semi-Supervised Pre-Training of Large-Scale Materials Models
This work addresses efficiency bottlenecks in computational materials science for researchers, though it is incremental as it builds on existing pre-training and fine-tuning approaches.
The paper tackles the computational expense of pre-training neural network potentials (NNPs) for materials science by proposing LaMM, a semi-supervised pre-training method that uses improved denoising self-supervised learning and load-balancing, leveraging a dataset of ~300 million semi-labeled samples to enhance fine-tuning speed and accuracy.
Neural network potentials (NNPs) are crucial for accelerating computational materials science by surrogating density functional theory (DFT) calculations. Improving their accuracy is possible through pre-training and fine-tuning, where an NNP model is first pre-trained on a large-scale dataset and then fine-tuned on a smaller target dataset. However, this approach is computationally expensive, mainly due to the cost of DFT-based dataset labeling and load imbalances during large-scale pre-training. To address this, we propose LaMM, a semi-supervised pre-training method incorporating improved denoising self-supervised learning and a load-balancing algorithm for efficient multi-node training. We demonstrate that our approach effectively leverages a large-scale dataset of $\sim$300 million semi-labeled samples to train a single NNP model, resulting in improved fine-tuning performance in terms of both speed and accuracy.