No Generation without Representation: Efficient Causal Protein Language Models Enable Zero-Shot Fitness Estimation
This addresses the problem for protein engineering practitioners by enabling efficient zero-shot fitness estimation and generation in a single model, though it is incremental in bridging existing architectures.
The paper tackles the divide between masked and causal protein language models by introducing Proust, a 309M-parameter causal model that achieves competitive fitness prediction (e.g., Spearman ρ=0.390 on ProteinGym) with much less compute and sets new state-of-the-art on indels.
Protein language models (PLMs) face a fundamental divide: masked language models (MLMs) excel at fitness prediction while causal models enable generation, forcing practitioners to maintain separate architectures. We introduce \textbf{Proust}, a 309M-parameter causal PLM that bridges this gap through architectural innovations adapted from recent LLM research, including grouped-query attention with shared K/V projections, cross-layer value residuals, and depthwise causal convolutions. Trained on 33B tokens in 40 B200 GPU-hours, Proust achieves Spearman $ρ= 0.390$ on ProteinGym substitutions, competitive with MLMs requiring 50--200$\times$ the compute. On indels, Proust sets a new state-of-the-art, outperforming models up to 20$\times$ larger. On EVEREST viral fitness benchmarks, it approaches structure-aware methods using sequence alone. These powerful representations position Proust in a sweet spot as it also retains native generative capabilities that MLMs lack by design. Interpretability analysis reveals that per-position entropy variance predicts, to an extent, when retrieval augmentation helps and hurts. Such insights can grow in both quantity and quality at scale and inform capabilities such as test-time scaling. Code and weights are available at https://github.com/Furkan9015/proust-inference