Adversarial Contrastive Pre-training for Protein Sequences
This work addresses protein sequence analysis for bioinformatics, but it is incremental as it adapts existing NLP advances to this domain.
The authors tackled the problem of pre-training protein sequence models by introducing an adversarial contrastive method, showing compelling improvements over traditional masked language model pre-training but noting significant computational costs.
Recent developments in Natural Language Processing (NLP) demonstrate that large-scale, self-supervised pre-training can be extremely beneficial for downstream tasks. These ideas have been adapted to other domains, including the analysis of the amino acid sequences of proteins. However, to date most attempts on protein sequences rely on direct masked language model style pre-training. In this work, we design a new, adversarial pre-training method for proteins, extending and specializing similar advances in NLP. We show compelling results in comparison to traditional MLM pre-training, though further development is needed to ensure the gains are worth the significant computational cost.