Metadata Might Make Language Models Better
This work addresses the challenge of better modeling historical texts for researchers and archivists, but it is incremental as it builds on existing time-masking methods.
The paper tackles the problem of improving language models for historical collections by incorporating metadata, finding that including temporal, political, and geographical information enhances performance on tasks like pseudo-perplexity and classification, leading to more robust and fairer models.
This paper discusses the benefits of including metadata when training language models on historical collections. Using 19th-century newspapers as a case study, we extend the time-masking approach proposed by Rosin et al., 2022 and compare different strategies for inserting temporal, political and geographical information into a Masked Language Model. After fine-tuning several DistilBERT on enhanced input data, we provide a systematic evaluation of these models on a set of evaluation tasks: pseudo-perplexity, metadata mask-filling and supervised classification. We find that showing relevant metadata to a language model has a beneficial impact and may even produce more robust and fairer models.