Out-of-Domain Evaluation of Finnish Dependency Parsing
This addresses the issue of parser robustness for real-world applications where data differs from training, specifically for Finnish language processing, but is incremental as it focuses on evaluation rather than new methods.
The paper tackles the problem of evaluating dependency parsers on out-of-domain data by introducing a new Finnish out-of-domain treebank with 19,382 words from diverse sources, and shows it creates a more challenging evaluation setting compared to existing treebanks.
The prevailing practice in the academia is to evaluate the model performance on in-domain evaluation data typically set aside from the training corpus. However, in many real world applications the data on which the model is applied may very substantially differ from the characteristics of the training data. In this paper, we focus on Finnish out-of-domain parsing by introducing a novel UD Finnish-OOD out-of-domain treebank including five very distinct data sources (web documents, clinical, online discussions, tweets, and poetry), and a total of 19,382 syntactic words in 2,122 sentences released under the Universal Dependencies framework. Together with the new treebank, we present extensive out-of-domain parsing evaluation utilizing the available section-level information from three different Finnish UD treebanks (TDT, PUD, OOD). Compared to the previously existing treebanks, the new Finnish-OOD is shown include sections more challenging for the general parser, creating an interesting evaluation setting and yielding valuable information for those applying the parser outside of its training domain.