Measuring and Modifying Factual Knowledge in Large Language Models
This work addresses the need for reliable knowledge measurement and modification in LLMs, which is crucial for downstream tasks, though it appears incremental as it builds on existing approaches.
The paper tackled the problem of measuring factual knowledge in large language models (LLMs) by proposing an information theory-based framework, achieving over 35% improvement in accuracy compared to previous methods in synthetic experiments. It also explored knowledge instillation methods and demonstrated applications for extracting unlearned and mislearned facts.
Large Language Models (LLMs) store an extensive amount of factual knowledge obtained from vast collections of text. To effectively utilize these models for downstream tasks, it is crucial to have reliable methods for measuring their knowledge. However, existing approaches for knowledge measurement have certain limitations, and despite recent efforts, they fail to provide accurate measurements and the necessary insights for modifying the knowledge within LLMs. In this work, we employ information theory-based measurements to provide a framework estimating the factual knowledge contained within large language models. More specifically, we measure knowledge by analyzing the LLM's prediction probability distribution before and after instilling the target knowledge, employing metrics such as entropy and KL-divergence. Introducing our metrics, we first assess their accuracy in comparison to previous ranking-based methods, surpassing them by over $35\%$ in a synthetic experiment. Then, we explore two prominent methods of knowledge instillation, discovering that LLMs exhibit limitations in capturing new knowledge under specific circumstances for one of these methods. Lastly, we demonstrate the applicability of our methods in extracting unlearned and mislearned facts in LLMs through their application to in-context learning. We make code and data for all methods and experiments in this paper publicly available.