Towards Data-Efficient Pretraining for Atomic Property Prediction
This addresses the challenge of computational and data efficiency for researchers and practitioners in atomic property prediction, offering an incremental improvement over existing methods.
The paper tackles the problem of data-inefficient pretraining for atomic property prediction by showing that pretraining on a smaller, task-relevant dataset can match or surpass large-scale pretraining, achieving similar performance with as little as 1/24th of the computational cost.
This paper challenges the recent paradigm in atomic property prediction that links progress to growing dataset sizes and computational resources. We show that pretraining on a carefully selected, task-relevant dataset can match or even surpass large-scale pretraining, while using as little as 1/24th of the computational cost. We introduce the Chemical Similarity Index (CSI), a novel metric inspired by computer vision's Fréchet Inception Distance, for molecular graphs which quantifies the alignment between upstream pretraining datasets and downstream tasks. By selecting the most relevant dataset with minimal CSI distance, we show that models pretrained on a smaller, focused dataset consistently outperform those pretrained on massive, mixed datasets such as JMP, even when those larger datasets include the relevant dataset. Counterintuitively, we also find that indiscriminately adding more data can degrade model performance when the additional data poorly aligns with the task at hand. Our findings highlight that quality often outperforms quantity in pretraining for atomic property prediction.