Viral Proteins Reveal Geometry of Protein Language Models
For researchers using protein language models, this work reveals that embeddings encode a general nativeness bias while retaining group-specific information, impacting interpretation of model representations.
Protein language models exhibit a dominant nativeness axis in embedding space that orders sequences from cellular to viral to random, with scaling contracting this axis unevenly across viral families. Despite this, viral proteins remain linearly separable beyond zero-shot perplexity and shallow sequence features.
Protein language models are trained on highly imbalanced datasets, raising the question of how they represent underrepresented biological sequences. Using viral proteins as a case study across ESM model families, we identify a dominant nativeness axis in embedding space, aligned with masked reconstruction perplexity, that orders sequences from well-modeled cellular proteins through viral proteins to shuffled and random sequences. Scaling contracts this axis unevenly across viral families. Despite this, protein language model embeddings retain viral-specific signal: viral proteins remain linearly separable beyond zero-shot perplexity and shallow sequence features. Together, these results suggest that pLM representations are structured by a general notion of nativeness while preserving information specific to distinct biological groups.