Data-Efficient Protein 3D Geometric Pretraining via Refinement of Diffused Protein Structure Decoy
This work addresses the problem of data-efficient protein representation learning for biological applications like drug design, though it appears incremental as it builds on existing pretraining paradigms.
The paper tackles the challenges of protein structure pretraining by introducing a 3D geometric-based pretext task called RefineDiff, which refines diffused protein structure decoys, and achieves comparable performance on downstream tasks using less than 1% of the data required by state-of-the-art models.
Learning meaningful protein representation is important for a variety of biological downstream tasks such as structure-based drug design. Having witnessed the success of protein sequence pretraining, pretraining for structural data which is more informative has become a promising research topic. However, there are three major challenges facing protein structure pretraining: insufficient sample diversity, physically unrealistic modeling, and the lack of protein-specific pretext tasks. To try to address these challenges, we present the 3D Geometric Pretraining. In this paper, we propose a unified framework for protein pretraining and a 3D geometric-based, data-efficient, and protein-specific pretext task: RefineDiff (Refine the Diffused Protein Structure Decoy). After pretraining our geometric-aware model with this task on limited data(less than 1% of SOTA models), we obtained informative protein representations that can achieve comparable performance for various downstream tasks.