Parsing with Pretrained Language Models, Multiple Datasets, and Dataset Embeddings
This work addresses the problem of improving multilingual dependency parsing by leveraging multiple datasets, though it is incremental as it builds on existing embedding methods with transformer models.
The study investigated the effectiveness of dataset embeddings in a transformer-based multilingual dependency parser, finding that embedding datasets at the encoder level yields the highest performance gains, particularly for small datasets or those with low baseline scores, and that training on all datasets combined performs similarly to clustering by language-relatedness.
With an increase of dataset availability, the potential for learning from a variety of data sources has increased. One particular method to improve learning from multiple data sources is to embed the data source during training. This allows the model to learn generalizable features as well as distinguishing features between datasets. However, these dataset embeddings have mostly been used before contextualized transformer-based embeddings were introduced in the field of Natural Language Processing. In this work, we compare two methods to embed datasets in a transformer-based multilingual dependency parser, and perform an extensive evaluation. We show that: 1) embedding the dataset is still beneficial with these models 2) performance increases are highest when embedding the dataset at the encoder level 3) unsurprisingly, we confirm that performance increases are highest for small datasets and datasets with a low baseline score. 4) we show that training on the combination of all datasets performs similarly to designing smaller clusters based on language-relatedness.