MELT: Materials-aware Continued Pre-training for Language Model Adaptation to Materials Science
This work addresses the challenge of efficiently adapting language models for materials science, which is incremental as it builds on existing continued pre-training strategies by incorporating domain-specific knowledge and training strategies.
The authors tackled the problem of adapting pre-trained language models to materials science by introducing MELT, a continued pre-training method that integrates a curriculum based on semantic graphs, resulting in superior performance over existing methods across diverse benchmarks.
We introduce a novel continued pre-training method, MELT (MatEriaLs-aware continued pre-Training), specifically designed to efficiently adapt the pre-trained language models (PLMs) for materials science. Unlike previous adaptation strategies that solely focus on constructing domain-specific corpus, MELT comprehensively considers both the corpus and the training strategy, given that materials science corpus has distinct characteristics from other domains. To this end, we first construct a comprehensive materials knowledge base from the scientific corpus by building semantic graphs. Leveraging this extracted knowledge, we integrate a curriculum into the adaptation process that begins with familiar and generalized concepts and progressively moves toward more specialized terms. We conduct extensive experiments across diverse benchmarks to verify the effectiveness and generality of MELT. A comprehensive evaluation convincingly supports the strength of MELT, demonstrating superior performance compared to existing continued pre-training methods. The in-depth analysis also shows that MELT enables PLMs to effectively represent materials entities compared to the existing adaptation methods, thereby highlighting its broad applicability across a wide spectrum of materials science.