Pareto Probing: Trading Off Accuracy for Complexity
This work addresses the need for more rigorous evaluation methods in NLP for probing word representations, though it is incremental as it builds on existing probing literature.
The authors tackled the problem of evaluating linguistic structure in word representations by proposing a probe metric based on the trade-off between complexity and performance, and found that common probing tasks are inadequate, leading them to propose full dependency parsing as a better task.
The question of how to probe contextual word representations for linguistic structure in a way that is both principled and useful has seen significant attention recently in the NLP literature. In our contribution to this discussion, we argue for a probe metric that reflects the fundamental trade-off between probe complexity and performance: the Pareto hypervolume. To measure complexity, we present a number of parametric and non-parametric metrics. Our experiments using Pareto hypervolume as an evaluation metric show that probes often do not conform to our expectations -- e.g., why should the non-contextual fastText representations encode more morpho-syntactic information than the contextual BERT representations? These results suggest that common, simplistic probing tasks, such as part-of-speech labeling and dependency arc labeling, are inadequate to evaluate the linguistic structure encoded in contextual word representations. This leads us to propose full dependency parsing as a probing task. In support of our suggestion that harder probing tasks are necessary, our experiments with dependency parsing reveal a wide gap in syntactic knowledge between contextual and non-contextual representations.