FootGPT : A Large Language Model Development Experiment on a Minimal Setting
This work addresses the challenge of creating domain-specific language models for soccer data analysis with constrained resources, but it is incremental as it applies existing fine-tuning techniques to a new dataset.
The authors tackled the problem of developing a specific-purpose language model for interpreting soccer data by fine-tuning a 1-billion-parameter general model with a curated dataset on Italian football league statistics, using low-rank adaptation and a short training duration to explore minimal resource settings.
With recent empirical observations, it has been argued that the most significant aspect of developing accurate language models may be the proper dataset content and training strategy compared to the number of neural parameters, training duration or dataset size. Following this argument, we opted to fine tune a one billion parameter size trained general purpose causal language model with a dataset curated on team statistics of the Italian football league first ten game weeks, using low rank adaptation. The limited training dataset was compiled based on a framework where a powerful commercial large language model provides distilled paragraphs and question answer pairs as intended. The training duration was kept relatively short to provide a basis for our minimal setting exploration. We share our key observations on the process related to developing a specific purpose language model which is intended to interpret soccer data with constrained resources in this article.