Unveiling Scaling Behaviors in Molecular Language Models: Effects of Model Size, Data, and Representation
This work addresses the problem of optimizing resource allocation for molecular generative models, which is important for researchers and developers in computational chemistry and drug discovery, though it is incremental in clarifying existing debates.
The study systematically investigated scaling laws in molecular language models by training 300 models and conducting over 10,000 experiments, revealing clear scaling patterns for both pretraining and downstream tasks and showing that molecular representation significantly impacts performance.
Molecular generative models, often employing GPT-style language modeling on molecular string representations, have shown promising capabilities when scaled to large datasets and model sizes. However, it remains unclear and subject to debate whether these models adhere to predictable scaling laws under fixed computational budgets, which is a crucial understanding for optimally allocating resources between model size, data volume, and molecular representation. In this study, we systematically investigate the scaling behavior of molecular language models across both pretraining and downstream tasks. We train 300 models and conduct over 10,000 experiments, rigorously controlling compute budgets while independently varying model size, number of training tokens, and molecular representation. Our results demonstrate clear scaling laws in molecular models for both pretraining and downstream transfer, reveal the substantial impact of molecular representation on performance, and explain previously observed inconsistencies in scaling behavior for molecular generation. Additionally, we publicly release the largest library of molecular language models to date to facilitate future research and development. Code and models are available at https://github.com/SZU-ADDG/MLM-Scaling.