DagoBERT: Generating Derivational Morphology with a Pretrained Language Model
This addresses the challenge of derivational morphology generation for natural language processing applications, but it is incremental as it builds on existing BERT models.
The study tackled the problem of whether pretrained language models can generate derivationally complex words, using BERT as an example, and found that their best model, DagoBERT, outperformed the previous state of the art in derivation generation.
Can pretrained language models (PLMs) generate derivationally complex words? We present the first study investigating this question, taking BERT as the example PLM. We examine BERT's derivational capabilities in different settings, ranging from using the unmodified pretrained model to full finetuning. Our best model, DagoBERT (Derivationally and generatively optimized BERT), clearly outperforms the previous state of the art in derivation generation (DG). Furthermore, our experiments show that the input segmentation crucially impacts BERT's derivational knowledge, suggesting that the performance of PLMs could be further improved if a morphologically informed vocabulary of units were used.