FeynTune: Large Language Models for High-Energy Theory
This work addresses the need for specialized language models in high-energy theoretical physics, though it is incremental as it builds on existing fine-tuning methods.
The researchers tackled the problem of improving language models for high-energy theoretical physics by fine-tuning variants of the Llama-3.1 model on arXiv abstracts, and they achieved performance gains over the base model on abstract completion tasks.
We present specialized Large Language Models for theoretical High-Energy Physics, obtained as 20 fine-tuned variants of the 8-billion parameter Llama-3.1 model. Each variant was trained on arXiv abstracts (through August 2024) from different combinations of hep-th, hep-ph and gr-qc. For a comparative study, we also trained models on datasets that contained abstracts from disparate fields such as the q-bio and cs categories. All models were fine-tuned using two distinct Low-Rank Adaptation fine-tuning approaches and varying dataset sizes, and outperformed the base model on hep-th abstract completion tasks. We compare performance against leading commercial LLMs (ChatGPT, Claude, Gemini, DeepSeek) and derive insights for further developing specialized language models for High-Energy Theoretical Physics.