A Comprehensive Evaluation of Semantic Relation Knowledge of Pretrained Language Models and Humans
This work addresses the incomplete understanding of PLMs' semantic relation knowledge for researchers in NLP, providing a comprehensive evaluation framework, though it is incremental as it extends existing research with new relations and metrics.
The paper tackled the problem of evaluating pretrained language models' (PLMs) knowledge of semantic relations beyond hypernymy, introducing a framework covering five relations and five metrics, and found a significant knowledge gap between humans and models, with antonymy being the only relation where models performed reasonably well.
Recently, much work has concerned itself with the enigma of what exactly pretrained language models~(PLMs) learn about different aspects of language, and how they learn it. One stream of this type of research investigates the knowledge that PLMs have about semantic relations. However, many aspects of semantic relations were left unexplored. Generally, only one relation has been considered, namely hypernymy. Furthermore, previous work did not measure humans' performance on the same task as that performed by the PLMs. This means that at this point in time, there is only an incomplete view of the extent of these models' semantic relation knowledge. To address this gap, we introduce a comprehensive evaluation framework covering five relations beyond hypernymy, namely hyponymy, holonymy, meronymy, antonymy, and synonymy. We use five metrics (two newly introduced here) for recently untreated aspects of semantic relation knowledge, namely soundness, completeness, symmetry, prototypicality, and distinguishability. Using these, we can fairly compare humans and models on the same task. Our extensive experiments involve six PLMs, four masked and two causal language models. The results reveal a significant knowledge gap between humans and models for all semantic relations. In general, causal language models, despite their wide use, do not always perform significantly better than masked language models. Antonymy is the outlier relation where all models perform reasonably well. The evaluation materials can be found at https://github.com/hancules/ProbeResponses.