Regression with Large Language Models for Materials and Molecular Property Prediction
This work shows LLMs can be applied to scientific regression tasks, potentially broadening their use in chemistry and materials science, but it is incremental as it adapts existing models to new data.
The authors tackled the problem of using large language models (LLMs) for material and molecular property regression, demonstrating that fine-tuned LLaMA 3 can rival standard models like random forests on the QM9 dataset, though its errors are 5-10x higher than state-of-the-art models.
We demonstrate the ability of large language models (LLMs) to perform material and molecular property regression tasks, a significant deviation from the conventional LLM use case. We benchmark the Large Language Model Meta AI (LLaMA) 3 on several molecular properties in the QM9 dataset and 24 materials properties. Only composition-based input strings are used as the model input and we fine tune on only the generative loss. We broadly find that LLaMA 3, when fine-tuned using the SMILES representation of molecules, provides useful regression results which can rival standard materials property prediction models like random forest or fully connected neural networks on the QM9 dataset. Not surprisingly, LLaMA 3 errors are 5-10x higher than those of the state-of-the-art models that were trained using far more granular representation of molecules (e.g., atom types and their coordinates) for the same task. Interestingly, LLaMA 3 provides improved predictions compared to GPT-3.5 and GPT-4o. This work highlights the versatility of LLMs, suggesting that LLM-like generative models can potentially transcend their traditional applications to tackle complex physical phenomena, thus paving the way for future research and applications in chemistry, materials science and other scientific domains.