GPT-SW3: An Autoregressive Language Model for the Nordic Languages
This addresses the problem of limited language resources for Nordic speakers and researchers, though it is incremental as it applies existing methods to new data.
The paper tackles the lack of large generative language models for Nordic languages by developing GPT-SW3, the first native model for these languages, covering the entire development process from data collection to evaluation.
This paper details the process of developing the first native large generative language model for the Nordic languages, GPT-SW3. We cover all parts of the development process, from data collection and processing, training configuration and instruction finetuning, to evaluation and considerations for release strategies. We hope that this paper can serve as a guide and reference for other researchers that undertake the development of large generative models for smaller languages.