Protein Language Models Diverge from Natural Language: Comparative Analysis and Improved Inference
This work addresses the need for more efficient and accurate protein function prediction for researchers in computational biology, though it is incremental as it adapts an existing technique from natural language processing.
The paper tackled the problem of transformer-based protein language models (PLMs) diverging from natural language models due to domain differences, and improved inference by adapting an early-exit technique to achieve accuracy gains of 0.4 to 7.01 percentage points and efficiency improvements over 10% on protein non-structural property prediction tasks.
Modern Protein Language Models (PLMs) apply transformer-based model architectures from natural language processing to biological sequences, predicting a variety of protein functions and properties. However, protein language has key differences from natural language, such as a rich functional space despite a vocabulary of only 20 amino acids. These differences motivate research into how transformer-based architectures operate differently in the protein domain and how we can better leverage PLMs to solve protein-related tasks. In this work, we begin by directly comparing how the distribution of information stored across layers of attention heads differs between the protein and natural language domain. Furthermore, we adapt a simple early-exit technique-originally used in the natural language domain to improve efficiency at the cost of performance-to achieve both increased accuracy and substantial efficiency gains in protein non-structural property prediction by allowing the model to automatically select protein representations from the intermediate layers of the PLMs for the specific task and protein at hand. We achieve performance gains ranging from 0.4 to 7.01 percentage points while simultaneously improving efficiency by over 10 percent across models and non-structural prediction tasks. Our work opens up an area of research directly comparing how language models change behavior when moved into the protein domain and advances language modeling in biological domains.