MolE: a molecular foundation model for drug discovery
This work addresses the challenge of generalization in drug discovery models for researchers and practitioners, though it is incremental as it adapts existing architectures to a specific domain.
The authors tackled the problem of predicting molecular properties with limited labeled data by developing MolE, a molecular foundation model that adapts DeBERTa for molecular graphs and uses a two-step pretraining strategy. The result is state-of-the-art performance on 9 out of 22 ADMET tasks in the Therapeutic Data Commons.
Models that accurately predict properties based on chemical structure are valuable tools in drug discovery. However, for many properties, public and private training sets are typically small, and it is difficult for the models to generalize well outside of the training data. Recently, large language models have addressed this problem by using self-supervised pretraining on large unlabeled datasets, followed by fine-tuning on smaller, labeled datasets. In this paper, we report MolE, a molecular foundation model that adapts the DeBERTa architecture to be used on molecular graphs together with a two-step pretraining strategy. The first step of pretraining is a self-supervised approach focused on learning chemical structures, and the second step is a massive multi-task approach to learn biological information. We show that fine-tuning pretrained MolE achieves state-of-the-art results on 9 of the 22 ADMET tasks included in the Therapeutic Data Commons.