TransPolymer: a Transformer-based language model for polymer property predictions
This work addresses the problem of expensive experimental or simulation-based polymer property evaluation for researchers in polymer science, though it is incremental as it applies an existing method to a new domain.
The authors tackled polymer property prediction by introducing TransPolymer, a Transformer-based language model, which achieved superior performance on ten benchmarks, benefiting from pretraining on large unlabeled data.
Accurate and efficient prediction of polymer properties is of great significance in polymer design. Conventionally, expensive and time-consuming experiments or simulations are required to evaluate polymer functions. Recently, Transformer models, equipped with self-attention mechanisms, have exhibited superior performance in natural language processing. However, such methods have not been investigated in polymer sciences. Herein, we report TransPolymer, a Transformer-based language model for polymer property prediction. Our proposed polymer tokenizer with chemical awareness enables learning representations from polymer sequences. Rigorous experiments on ten polymer property prediction benchmarks demonstrate the superior performance of TransPolymer. Moreover, we show that TransPolymer benefits from pretraining on large unlabeled dataset via Masked Language Modeling. Experimental results further manifest the important role of self-attention in modeling polymer sequences. We highlight this model as a promising computational tool for promoting rational polymer design and understanding structure-property relationships from a data science view.