Pre-Training of Deep Bidirectional Protein Sequence Representations with Structural Information
This work addresses the challenge of leveraging unlabeled protein data for biologists by providing a more effective pre-training method, though it is incremental as it builds on existing semi-supervised learning approaches.
The paper tackles the performance limitations of protein sequence pre-training methods that rely solely on language modeling by introducing PLUS, a novel pre-training scheme that incorporates structural information through masked language modeling and same-family prediction. The resulting PLUS-RNN model outperforms similar-sized models in six out of seven protein biology tasks, demonstrating improved effectiveness.
Bridging the exponentially growing gap between the numbers of unlabeled and labeled protein sequences, several studies adopted semi-supervised learning for protein sequence modeling. In these studies, models were pre-trained with a substantial amount of unlabeled data, and the representations were transferred to various downstream tasks. Most pre-training methods solely rely on language modeling and often exhibit limited performance. In this paper, we introduce a novel pre-training scheme called PLUS, which stands for Protein sequence representations Learned Using Structural information. PLUS consists of masked language modeling and a complementary protein-specific pre-training task, namely same-family prediction. PLUS can be used to pre-train various model architectures. In this work, we use PLUS to pre-train a bidirectional recurrent neural network and refer to the resulting model as PLUS-RNN. Our experiment results demonstrate that PLUS-RNN outperforms other models of similar size solely pre-trained with the language modeling in six out of seven widely used protein biology tasks. Furthermore, we present the results from our qualitative interpretation analyses to illustrate the strengths of PLUS-RNN. PLUS provides a novel way to exploit evolutionary relationships among unlabeled proteins and is broadly applicable across a variety of protein biology tasks. We expect that the gap between the numbers of unlabeled and labeled proteins will continue to grow exponentially, and the proposed pre-training method will play a larger role.