RobBERTje: a Distilled Dutch BERT Model
This work provides more efficient and less biased models for Dutch natural language processing tasks, though it is incremental as it builds on existing distillation methods.
The authors tackled the resource-intensive deployment and fine-tuning of large Dutch language models by creating distilled versions called RobBERTje, finding that shuffled vs. non-shuffled datasets yield similar performance, merged sentences improve training speed and long-sequence tasks, and distilled models reduce gender-stereotypical bias compared to the teacher model.
Pre-trained large-scale language models such as BERT have gained a lot of attention thanks to their outstanding performance on a wide range of natural language tasks. However, due to their large number of parameters, they are resource-intensive both to deploy and to fine-tune. Researchers have created several methods for distilling language models into smaller ones to increase efficiency, with a small performance trade-off. In this paper, we create several different distilled versions of the state-of-the-art Dutch RobBERT model and call them RobBERTje. The distillations differ in their distillation corpus, namely whether or not they are shuffled and whether they are merged with subsequent sentences. We found that the performance of the models using the shuffled versus non-shuffled datasets is similar for most tasks and that randomly merging subsequent sentences in a corpus creates models that train faster and perform better on tasks with long sequences. Upon comparing distillation architectures, we found that the larger DistilBERT architecture worked significantly better than the Bort hyperparametrization. Interestingly, we also found that the distilled models exhibit less gender-stereotypical bias than its teacher model. Since smaller architectures decrease the time to fine-tune, these models allow for more efficient training and more lightweight deployment of many Dutch downstream language tasks.