Profile Prediction: An Alignment-Based Pre-Training Task for Protein Sequence Models
This work addresses the challenge of effective pre-training for protein sequence models, which is crucial for leveraging abundant unlabeled protein data to improve predictions for biologists and drug discovery researchers. It offers an incremental improvement over existing NLP-inspired methods.
This paper introduces a new pre-training task for protein sequence models called profile prediction, which involves directly predicting protein profiles from multiple sequence alignments. The authors demonstrate that this method, combined with a multi-task objective, outperforms masked language modeling on all five standardized downstream protein prediction tasks.
For protein sequence datasets, unlabeled data has greatly outpaced labeled data due to the high cost of wet-lab characterization. Recent deep-learning approaches to protein prediction have shown that pre-training on unlabeled data can yield useful representations for downstream tasks. However, the optimal pre-training strategy remains an open question. Instead of strictly borrowing from natural language processing (NLP) in the form of masked or autoregressive language modeling, we introduce a new pre-training task: directly predicting protein profiles derived from multiple sequence alignments. Using a set of five, standardized downstream tasks for protein models, we demonstrate that our pre-training task along with a multi-task objective outperforms masked language modeling alone on all five tasks. Our results suggest that protein sequence models may benefit from leveraging biologically-inspired inductive biases that go beyond existing language modeling techniques in NLP.