Probing Chemical Language Models: Effects of Pre-training and Fine-tuning
For researchers using chemical language models, this work provides insights into which molecular features are encoded and how training affects them, though the findings are incremental.
The study probes 78 molecular substructures across 14 chemical language models to understand how pre-training and fine-tuning affect learned representations. Results show pre-training improves structure awareness, especially in upper layers, and fine-tuning selectively enhances task-relevant substructures.
Chemical language models (CLMs) are trained with linearized representations such as SMILES, yet it remains unclear which chemically meaningful substructures they encode. To foster a better understanding of CLMs, we conduct a systematic study and probe for 78 molecular substructures across eight pre-trained and six randomly initialized models. We furthermore study how fine-tuning on chemical downstream tasks affects the learned representations of molecular substructures. Our results show that pre-training generally improves molecular structure awareness of CLMs, particularly in the upper layers. Moreover, randomly initialized models already encode ring structures well in the first layer. Our analysis on two chemical downstream tasks further reveals that, interestingly, fine-tuning affects task-relevant molecular substructures more than others, indicating that the changes in the representations follow chemical theory.