Self Distillation Fine-Tuning of Protein Language Models Improves Versatility in Protein Design
This work addresses the problem of limited annotated data for protein design, enabling more versatile and reliable protein generation for researchers in bioengineering and synthetic biology, though it is incremental as it builds on existing fine-tuning methods.
The authors tackled the challenge of supervised fine-tuning for protein language models by introducing a self-distillation method that uses the model's own outputs with domain-specific filters to create training data, resulting in generated protein sequences that are more novel and exhibit improved stability and functionality, such as in the tryptophan synthase enzyme family.
Supervised fine-tuning (SFT) is a standard approach for adapting large language models to specialized domains, yet its application to protein sequence modeling and protein language models (PLMs) remains ad hoc. This is in part because high-quality annotated data are far more difficult to obtain for proteins than for natural language. We present a simple and general recipe for fast SFT of PLMs, designed to improve the fidelity, reliability, and novelty of generated protein sequences. Unlike existing approaches that require costly precompiled experimental datasets for SFT, our method leverages the PLM itself, integrating a lightweight curation pipeline with domain-specific filters to construct high-quality training data. These filters can independently refine a PLM's output and identify candidates for in vitro evaluation; when combined with SFT, they enable PLMs to generate more stable and functional enzymes, while expanding exploration into protein sequence space beyond natural variants. Although our approach is agnostic to both the choice of protein language model (PLM) and the protein system, we demonstrate its effectiveness with a genome-scale PLM (GenSLM) applied to the tryptophan synthase enzyme family. The supervised fine-tuned model generates sequences that are not only more novel but also display improved characteristics across both targeted design constraints and emergent protein property measures.