Chemical transformer compression for accelerating both training and inference of molecular modeling
This work addresses the high hardware requirements for training and inference in molecular modeling, offering a more efficient solution for researchers in drug and material design, though it is incremental as it builds on existing compression techniques.
The authors tackled the problem of large transformer models in molecular science by introducing DeLiCaTe, a compressed model using cross-layer parameter sharing and knowledge distillation, achieving a 4x faster training and inference rate with comparable performance in QSAR and VS modeling.
Transformer models have been developed in molecular science with excellent performance in applications including quantitative structure-activity relationship (QSAR) and virtual screening (VS). Compared with other types of models, however, they are large, which results in a high hardware requirement to abridge time for both training and inference processes. In this work, cross-layer parameter sharing (CLPS), and knowledge distillation (KD) are used to reduce the sizes of transformers in molecular science. Both methods not only have competitive QSAR predictive performance as compared to the original BERT model, but also are more parameter efficient. Furthermore, by integrating CLPS and KD into a two-state chemical network, we introduce a new deep lite chemical transformer model, DeLiCaTe. DeLiCaTe captures general-domains as well as task-specific knowledge, which lead to a 4x faster rate of both training and inference due to a 10- and 3-times reduction of the number of parameters and layers, respectively. Meanwhile, it achieves comparable performance in QSAR and VS modeling. Moreover, we anticipate that the model compression strategy provides a pathway to the creation of effective generative transformer models for organic drug and material design.