Pre-training via Denoising for Molecular Property Prediction
This work addresses data scarcity in molecular property prediction, an important problem for computational chemistry and drug discovery, with incremental improvements over existing methods.
The paper tackles the challenge of limited data in molecular property prediction from 3D structures by introducing a denoising-based pre-training technique that learns representations from large datasets of equilibrium structures, achieving a new state-of-the-art on most targets in the QM9 dataset.
Many important problems involving molecular property prediction from 3D structures have limited data, posing a generalization challenge for neural networks. In this paper, we describe a pre-training technique based on denoising that achieves a new state-of-the-art in molecular property prediction by utilizing large datasets of 3D molecular structures at equilibrium to learn meaningful representations for downstream tasks. Relying on the well-known link between denoising autoencoders and score-matching, we show that the denoising objective corresponds to learning a molecular force field -- arising from approximating the Boltzmann distribution with a mixture of Gaussians -- directly from equilibrium structures. Our experiments demonstrate that using this pre-training objective significantly improves performance on multiple benchmarks, achieving a new state-of-the-art on the majority of targets in the widely used QM9 dataset. Our analysis then provides practical insights into the effects of different factors -- dataset sizes, model size and architecture, and the choice of upstream and downstream datasets -- on pre-training.