The Fragility of Multi-Treebank Parsing Evaluation
This addresses methodological issues in NLP evaluation for researchers, but it is incremental as it builds on existing concerns about data selection.
The paper tackles the problem of biased treebank selection in parsing evaluation, showing that evaluating on single subsets leads to weak conclusions with substantial score variability across random subsets.
Treebank selection for parsing evaluation and the spurious effects that might arise from a biased choice have not been explored in detail. This paper studies how evaluating on a single subset of treebanks can lead to weak conclusions. First, we take a few contrasting parsers, and run them on subsets of treebanks proposed in previous work, whose use was justified (or not) on criteria such as typology or data scarcity. Second, we run a large-scale version of this experiment, create vast amounts of random subsets of treebanks, and compare on them many parsers whose scores are available. The results show substantial variability across subsets and that although establishing guidelines for good treebank selection is hard, it is possible to detect potentially harmful strategies.