CrystalFormer-RL: Reinforcement Fine-Tuning for Materials Design
This work addresses materials design for researchers and engineers, offering an incremental improvement by adapting existing reinforcement fine-tuning methods to a new domain.
The paper tackled the problem of designing materials with desirable yet conflicting properties by applying reinforcement fine-tuning to a generative model, resulting in enhanced stability and successful discovery of crystals with substantial dielectric constant and band gap simultaneously.
Reinforcement fine-tuning played an instrumental role in enhancing the instruction-following and reasoning abilities of large language models. In this work, we employ reinforcement fine-tuning for materials design, in which discriminative machine learning models are used to provide rewards to the autoregressive transformer-based materials generative model CrystalFormer. By optimizing the reward signals-such as energy above the convex hull and material properties figures of merit-reinforcement fine-tuning infuses knowledge from discriminative models into generative models. The resulting model, CrystalFormer-RL, shows enhanced stability in generated crystals and successfully discovers crystals with desirable yet conflicting material properties, such as substantial dielectric constant and band gap simultaneously. Notably, we observe that reinforcement fine-tuning not only enables the property-guided material design but also unlocks property-based material retrieval behavior of pretrained generative model. The present framework opens an exciting gateway to the synergies of the machine learning ecosystem for materials design.