Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly
This work addresses the issue of evaluating factual knowledge in language models for researchers in NLP, but it is incremental as it builds on prior probing methods.
The study tackled the problem of analyzing factual knowledge in pretrained language models by introducing negation and mispriming probing tasks, finding that models fail to distinguish negated from non-negated statements and are easily distracted by misprimes.
Building on Petroni et al. (2019), we propose two new probing tasks analyzing factual knowledge stored in Pretrained Language Models (PLMs). (1) Negation. We find that PLMs do not distinguish between negated ("Birds cannot [MASK]") and non-negated ("Birds can [MASK]") cloze questions. (2) Mispriming. Inspired by priming methods in human psychology, we add "misprimes" to cloze questions ("Talk? Birds can [MASK]"). We find that PLMs are easily distracted by misprimes. These results suggest that PLMs still have a long way to go to adequately learn human-like factual knowledge.