Enhancing Fact Retrieval in PLMs through Truthfulness
This work addresses the challenge of extracting accurate facts from PLMs for applications like soft knowledge bases, though it is incremental as it builds on existing methods for truthfulness assessment.
The paper tackles the problem of improving factual knowledge retrieval from pre-trained language models by using a helper model that assesses input truthfulness based on hidden states, resulting in up to a 33% enhancement in fact retrieval.
Pre-trained Language Models (PLMs) encode various facts about the world at their pre-training phase as they are trained to predict the next or missing word in a sentence. There has a been an interest in quantifying and improving the amount of facts that can be extracted from PLMs, as they have been envisioned to act as soft knowledge bases, which can be queried in natural language. Different approaches exist to enhance fact retrieval from PLM. Recent work shows that the hidden states of PLMs can be leveraged to determine the truthfulness of the PLMs' inputs. Leveraging this finding to improve factual knowledge retrieval remains unexplored. In this work, we investigate the use of a helper model to improve fact retrieval. The helper model assesses the truthfulness of an input based on the corresponding hidden states representations from the PLMs. We evaluate this approach on several masked PLMs and show that it enhances fact retrieval by up to 33\%. Our findings highlight the potential of hidden states representations from PLMs in improving their factual knowledge retrieval.