Operationalizing a National Digital Library: The Case for a Norwegian Transformer Model
This demonstrates that national libraries can create effective language models from noisy OCR data, potentially enabling similar efforts elsewhere.
The researchers built a Norwegian BERT model using digitized library collections, which outperforms multilingual BERT on Norwegian classification tasks and improves performance for some other languages in the corpus.
In this work, we show the process of building a large-scale training set from digital and digitized collections at a national library. The resulting Bidirectional Encoder Representations from Transformers (BERT)-based language model for Norwegian outperforms multilingual BERT (mBERT) models in several token and sequence classification tasks for both Norwegian Bokmål and Norwegian Nynorsk. Our model also improves the mBERT performance for other languages present in the corpus such as English, Swedish, and Danish. For languages not included in the corpus, the weights degrade moderately while keeping strong multilingual properties. Therefore, we show that building high-quality models within a memory institution using somewhat noisy optical character recognition (OCR) content is feasible, and we hope to pave the way for other memory institutions to follow.