Enabling LLM Knowledge Analysis via Extensive Materialization
This addresses the issue of availability bias in LLM knowledge analysis for researchers, providing a more objective and extensive dataset, though it is incremental in automating and scaling up existing analysis approaches.
The paper tackles the problem of analyzing large language models' factual knowledge by proposing a novel methodology to materialize it comprehensively, resulting in GPTKB, a knowledge base with 101 million relational triples for over 2.9 million entities from GPT-4o-mini, enabling simultaneous analysis of scale, accuracy, bias, cutoff, and consistency.
Large language models (LLMs) have majorly advanced NLP and AI, and next to their ability to perform a wide range of procedural tasks, a major success factor is their internalized factual knowledge. Since Petroni et al. (2019), analyzing this knowledge has gained attention. However, most approaches investigate one question at a time via modest-sized pre-defined samples, introducing an ``availability bias'' (Tversky&Kahnemann, 1973) that prevents the analysis of knowledge (or beliefs) of LLMs beyond the experimenter's predisposition. To address this challenge, we propose a novel methodology to comprehensively materialize an LLM's factual knowledge through recursive querying and result consolidation. Our approach is a milestone for LLM research, for the first time providing constructive insights into the scope and structure of LLM knowledge (or beliefs). As a prototype, we build GPTKB, a knowledge base (KB) comprising 101 million relational triples for over 2.9 million entities from GPT-4o-mini. We use GPTKB to exemplarily analyze GPT-4o-mini's factual knowledge in terms of scale, accuracy, bias, cutoff and consistency, at the same time. GPTKB is accessible at https://gptkb.org