Constructing Taxonomies from Pretrained Language Models
This addresses the challenge of automated taxonomy construction for NLP applications, representing an incremental advance with specific performance gains.
The paper tackles the problem of constructing taxonomic trees like WordNet using pretrained language models, achieving a 66.7 ancestor F1 score, which is a 20.0% relative improvement over prior methods, and extends results to nine other languages.
We present a method for constructing taxonomic trees (e.g., WordNet) using pretrained language models. Our approach is composed of two modules, one that predicts parenthood relations and another that reconciles those predictions into trees. The parenthood prediction module produces likelihood scores for each potential parent-child pair, creating a graph of parent-child relation scores. The tree reconciliation module treats the task as a graph optimization problem and outputs the maximum spanning tree of this graph. We train our model on subtrees sampled from WordNet, and test on non-overlapping WordNet subtrees. We show that incorporating web-retrieved glosses can further improve performance. On the task of constructing subtrees of English WordNet, the model achieves 66.7 ancestor F1, a 20.0% relative increase over the previous best published result on this task. In addition, we convert the original English dataset into nine other languages using Open Multilingual WordNet and extend our results across these languages.