Efficient Training of Transformers for Molecule Property Prediction on Small-scale Datasets
This work addresses the need for efficient computational methods to predict drug permeability across the blood-brain barrier, which is crucial for drug development but incremental in its approach.
The paper tackled the problem of predicting blood-brain barrier permeability for drug screening by proposing a GPS Transformer with Self Attention, achieving a state-of-the-art ROC-AUC of 78.8% on the BBBP dataset, which is a 5.5% improvement over existing models.
The blood-brain barrier (BBB) serves as a protective barrier that separates the brain from the circulatory system, regulating the passage of substances into the central nervous system. Assessing the BBB permeability of potential drugs is crucial for effective drug targeting. However, traditional experimental methods for measuring BBB permeability are challenging and impractical for large-scale screening. Consequently, there is a need to develop computational approaches to predict BBB permeability. This paper proposes a GPS Transformer architecture augmented with Self Attention, designed to perform well in the low-data regime. The proposed approach achieved a state-of-the-art performance on the BBB permeability prediction task using the BBBP dataset, surpassing existing models. With a ROC-AUC of 78.8%, the approach sets a state-of-the-art by 5.5%. We demonstrate that standard Self Attention coupled with GPS transformer performs better than other variants of attention coupled with GPS Transformer.