AstroMLab 2: AstroLLaMA-2-70B Model and Benchmarking Specialised LLMs for Astronomy
This research addresses the lack of objective evaluation for specialized LLMs in astronomy, providing a quantitative assessment for researchers and developers in the field. It is an incremental step in understanding domain-specific LLM performance.
The study evaluates specialized large language models (LLMs) for astronomy using a new benchmark of multiple-choice questions. It finds that smaller AstroLLaMA models underperform their base LLaMA-2-7B counterparts, but this can be partially mitigated by using high-quality pretraining data like arXiv summaries. For the 70B model, continual pretraining yields significant improvements, though supervised fine-tuning data remains a bottleneck.
Continual pretraining of large language models on domain-specific data has been proposed to enhance performance on downstream tasks. In astronomy, the previous absence of astronomy-focused benchmarks has hindered objective evaluation of these specialized LLM models. Leveraging a recent initiative to curate high-quality astronomical MCQs, this study aims to quantitatively assess specialized LLMs in astronomy. We find that the previously released AstroLLaMA series, based on LLaMA-2-7B, underperforms compared to the base model. We demonstrate that this performance degradation can be partially mitigated by utilizing high-quality data for continual pretraining, such as summarized text from arXiv. Despite the observed catastrophic forgetting in smaller models, our results indicate that continual pretraining on the 70B model can yield significant improvements. However, the current supervised fine-tuning dataset still constrains the performance of instruct models. In conjunction with this study, we introduce a new set of models, AstroLLaMA-3-8B and AstroLLaMA-2-70B, building upon the previous AstroLLaMA series.