SPE: Symmetrical Prompt Enhancement for Fact Probing
This work addresses the need for more accurate factual knowledge extraction from language models, which is incremental as it builds on existing prompt-based probing techniques.
The paper tackled the problem of probing pretrained language models for factual knowledge by addressing the symmetry between subject and object prediction, and the result was a significant improvement over previous methods on the LAMA dataset.
Pretrained language models (PLMs) have been shown to accumulate factual knowledge during pretrainingng (Petroni et al., 2019). Recent works probe PLMs for the extent of this knowledge through prompts either in discrete or continuous forms. However, these methods do not consider symmetry of the task: object prediction and subject prediction. In this work, we propose Symmetrical Prompt Enhancement (SPE), a continuous prompt-based method for factual probing in PLMs that leverages the symmetry of the task by constructing symmetrical prompts for subject and object prediction. Our results on a popular factual probing dataset, LAMA, show significant improvement of SPE over previous probing methods.