BERTese: Learning to Speak to BERT
This work addresses the challenge of efficiently extracting knowledge from language models for researchers and practitioners, representing an incremental improvement over existing methods.
The paper tackles the problem of extracting knowledge from pre-trained language models by automatically rewriting queries into 'BERTese', which is optimized for better knowledge extraction, and shows that this approach outperforms competing baselines, obviating the need for complex pipelines.
Large pre-trained language models have been shown to encode large amounts of world and commonsense knowledge in their parameters, leading to substantial interest in methods for extracting that knowledge. In past work, knowledge was extracted by taking manually-authored queries and gathering paraphrases for them using a separate pipeline. In this work, we propose a method for automatically rewriting queries into "BERTese", a paraphrase query that is directly optimized towards better knowledge extraction. To encourage meaningful rewrites, we add auxiliary loss functions that encourage the query to correspond to actual language tokens. We empirically show our approach outperforms competing baselines, obviating the need for complex pipelines. Moreover, BERTese provides some insight into the type of language that helps language models perform knowledge extraction.