Protein Language Model Zero-Shot Fitness Predictions are Improved by Inference-only Dropout
This provides a simple, training-free method to improve protein fitness prediction for computational biology applications, though it is incremental.
The authors tackled the problem of improving zero-shot fitness predictions from protein language models by showing that adding inference-only dropout between the embedding and transformer layers boosts performance on ProteinGym benchmarks, with a dropout rate of 0.1 consistently enhancing results across models.
Protein Language Models (PLMs) such as ESM2 have been shown to be capable of zero-shot prediction of critical scalar properties of proteins (fitness). In this work, we show that injecting a dropout layer at inference time between a PLM's featurizer/embedding layer and its transformer, and averaging its output akin to Monte-Carlo dropout increases zero-shot performance on a subset of the ProteinGym dataset. This is the case even when the model was not trained with dropouts to begin with, and does not require retraining or finetuning of the PLM. A dropout of 0.1 seems performant across all models.