Enhancing Protein Representation Learning via Manifold Restore Mixing
For researchers in protein representation learning, this work provides a simple yet effective data augmentation method that balances structural preservation and diversity, though it is an incremental improvement over existing mixup-based approaches.
The paper addresses the problem that existing data augmentation methods for protein representation learning either disrupt protein structure or reduce structural diversity. The proposed Manifold Restore Mixing (MRM) method mixes hidden representations of original and augmented data to restore lost structural information while introducing variations, achieving improved performance across various backbones and tasks.
Data augmentation (DA) has been proven to be an effective means for improving protein representation learning (PRL) by generating additional training samples. Although mainstream perturbation- and sampling-based augmentation methods can produce data containing sufficient variations, they carry the risk of disrupting the protein structure and function. Some crafted protein homology modeling tools can generate conformations, but reduce structural diversity. The above dilemmas lead us to a question: Can we restore the disrupted structure caused by DA operations, providing data with both the original structure and diverse variations? In this work, we first analyze and empirically reveal the structure defect and performance degradation issues of existing DA methods. Based on the findings, we propose a simple yet effective DA method, Manifold Restore Mixing (MRM), for protein representation learning. Specifically, inspired by manifold mixup, we mix the hidden representations of original and augmented protein data to generate new samples that restore structural information lost in DA while introducing diverse variations. Furthermore, we develop a sample difficulty scheduler that adjusts the beta distribution in mixup to provide models with progressively challenging mixed samples during training, which improves the final performance. Comprehensive experiments on various PRL backbones and downstream tasks demonstrate the effectiveness and generalization of our method. The complete code and weights will be released upon acceptance. We provide a implementation at https://github.com/KingGugu/MRM.