A Diversity Diet for a Healthier Model: A Case Study of French ModernBERT
This addresses the problem of reducing computational costs for NLP practitioners by optimizing pre-training data selection, though it is incremental as it builds on existing transformer models.
The study investigated the impact of diversity-driven sampling on ModernBERT pre-training, aiming to reduce dataset size while maintaining performance. It found that diversity-driven sampling improved performance by up to 10 points in some tasks and achieved comparable results with a 150M token dataset in 483 hours versus a 2.4B token dataset in 1,775 hours.
Diversity has been gaining interest in the NLP community in recent years. At the same time, state-of-the-art transformer models such as ModernBERT use very large pre-training datasets, which are driven by size rather than by diversity. This summons for an investigation of the impact of diversity on the ModernBERT pre-training. We do so in this study, with the express intent of reducing pre-training dataset size, while retaining at least comparable performance. We compare diversity-driven sampling algorithms, so as to pick the best one. We find that diversity-driven sampling allows in some tasks to gain 10 points relative to randomly-sampled pre-training data of commensurate size. We also see that a model pre-trained for 483h on a diversity-driven dataset of 150M tokens can yield a commensurate performance to a model pre-trained for 1,775h on a randomly-driven dataset of 2.4B tokens.