Using Distributional Principles for the Semantic Study of Contextual Language Models
This work addresses a gap in semantic analysis for NLP researchers, but it is incremental as it builds on existing probing methods.
The authors tackled the problem of understanding semantic similarity in contextual language models by applying distributional principles, specifically substitution, in controlled and open settings, revealing differences between static and contextual models.
Many studies were recently done for investigating the properties of contextual language models but surprisingly, only a few of them consider the properties of these models in terms of semantic similarity. In this article, we first focus on these properties for English by exploiting the distributional principle of substitution as a probing mechanism in the controlled context of SemCor and WordNet paradigmatic relations. Then, we propose to adapt the same method to a more open setting for characterizing the differences between static and contextual language models.