xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein
This addresses the need for versatile foundation models in protein science that can handle both understanding and generation tasks concurrently, representing a substantial advance rather than an incremental improvement.
The authors tackled the problem of protein language models being limited to either understanding or generation tasks by proposing xTrimoPGLM, a unified 100B-parameter model that simultaneously handles both types of tasks, achieving significant performance improvements in 18 protein understanding benchmarks and enabling de novo protein sequence generation.
Protein language models have shown remarkable success in learning biological information from protein sequences. However, most existing models are limited by either autoencoding or autoregressive pre-training objectives, which makes them struggle to handle protein understanding and generation tasks concurrently. We propose a unified protein language model, xTrimoPGLM, to address these two types of tasks simultaneously through an innovative pre-training framework. Our key technical contribution is an exploration of the compatibility and the potential for joint optimization of the two types of objectives, which has led to a strategy for training xTrimoPGLM at an unprecedented scale of 100 billion parameters and 1 trillion training tokens. Our extensive experiments reveal that 1) xTrimoPGLM significantly outperforms other advanced baselines in 18 protein understanding benchmarks across four categories. The model also facilitates an atomic-resolution view of protein structures, leading to an advanced 3D structural prediction model that surpasses existing language model-based tools. 2) xTrimoPGLM not only can generate de novo protein sequences following the principles of natural ones, but also can perform programmable generation after supervised fine-tuning (SFT) on curated sequences. These results highlight the substantial capability and versatility of xTrimoPGLM in understanding and generating protein sequences, contributing to the evolving landscape of foundation models in protein science.