Ensemble Knowledge Distillation for Machine Learning Interatomic Potentials
This work addresses the challenge of developing more accurate and stable interatomic potentials for computational chemistry and materials science, representing an incremental improvement in a domain-specific context.
The authors tackled the problem of training accurate machine learning interatomic potentials (MLIPs) with limited high-fidelity quantum chemistry data, often lacking energy gradients, by proposing an ensemble knowledge distillation method. The result was a student MLIP that achieved new state-of-the-art accuracy on the COMP6 benchmark and improved stability in molecular dynamics simulations.
The quality of machine learning interatomic potentials (MLIPs) strongly depends on the quantity of training data as well as the quantum chemistry (QC) level of theory used. Datasets generated with high-fidelity QC methods are typically restricted to small molecules and may be missing energy gradients, which make it difficult to train accurate MLIPs. We present an ensemble knowledge distillation (EKD) method to improve MLIP accuracy when trained to energy-only datasets. First, multiple teacher models are trained to QC energies and then generate atomic forces for all configurations in the dataset. Next, the student MLIP is trained to both QC energies and to ensemble-averaged forces generated by the teacher models. We apply this workflow on the ANI-1ccx dataset where the configuration energies computed at the coupled cluster level of theory. The resulting student MLIPs achieve new state-of-the-art accuracy on the COMP6 benchmark and show improved stability for molecular dynamics simulations.