SFM-Protein: Integrative Co-evolutionary Pre-training for Advanced Protein Sequence Representation
This addresses the problem of limited co-evolutionary capture in protein modeling for researchers, representing an incremental improvement over existing methods.
The paper tackled the challenge of capturing co-evolutionary information in protein sequences by introducing an integrative pre-training strategy for protein foundation models, resulting in superior generalization and outperforming baselines like ESM across diverse downstream tasks.
Proteins, essential to biological systems, perform functions intricately linked to their three-dimensional structures. Understanding the relationship between protein structures and their amino acid sequences remains a core challenge in protein modeling. While traditional protein foundation models benefit from pre-training on vast unlabeled datasets, they often struggle to capture critical co-evolutionary information, which evolutionary-based methods excel at. In this study, we introduce a novel pre-training strategy for protein foundation models that emphasizes the interactions among amino acid residues to enhance the extraction of both short-range and long-range co-evolutionary features from sequence data. Trained on a large-scale protein sequence dataset, our model demonstrates superior generalization ability, outperforming established baselines of similar size, including the ESM model, across diverse downstream tasks. Experimental results confirm the model's effectiveness in integrating co-evolutionary information, marking a significant step forward in protein sequence-based modeling.