The Nordic Pile: A 1.2TB Nordic Dataset for Language Modeling
This dataset addresses the problem of data scarcity for building LLMs in smaller languages, specifically for Nordic language communities, but is incremental as it focuses on data curation rather than novel methods.
The authors tackled the challenge of limited text corpora for Nordic languages by curating a high-quality 1.2TB dataset in Danish, Icelandic, Norwegian, Swedish, and English to facilitate LLM development.
Pre-training Large Language Models (LLMs) require massive amounts of text data, and the performance of the LLMs typically correlates with the scale and quality of the datasets. This means that it may be challenging to build LLMs for smaller languages such as Nordic ones, where the availability of text corpora is limited. In order to facilitate the development of the LLMS in the Nordic languages, we curate a high-quality dataset consisting of 1.2TB of text, in all of the major North Germanic languages (Danish, Icelandic, Norwegian, and Swedish), as well as some high-quality English data. This paper details our considerations and processes for collecting, cleaning, and filtering the dataset.