Decoding Knowledge in Large Language Models: A Framework for Categorization and Comprehension
This work addresses the problem of nuanced knowledge evaluation in LLMs for researchers, though it appears incremental as it builds on existing techniques like prompting and reinforcement learning.
The paper tackles the challenge of understanding how large language models (LLMs) acquire and apply knowledge by introducing the K-(CSA)^2 framework, which categorizes knowledge based on correctness and confidence, revealing that chain-of-thought prompting enhances performance and higher layers encode more high-confidence knowledge.
Understanding how large language models (LLMs) acquire, retain, and apply knowledge remains an open challenge. This paper introduces a novel framework, K-(CSA)^2, which categorizes LLM knowledge along two dimensions: correctness and confidence. The framework defines six categories of knowledge, ranging from highly confident correctness to confidently held misconceptions, enabling a nuanced evaluation of model comprehension beyond binary accuracy. Using this framework, we demonstrate how techniques like chain-of-thought prompting and reinforcement learning with human feedback fundamentally alter the knowledge structures of internal (pre-trained) and external (context-dependent) knowledge in LLMs. CoT particularly enhances base model performance and shows synergistic benefits when applied to aligned LLMs. Moreover, our layer-wise analysis reveals that higher layers in LLMs encode more high-confidence knowledge, while low-confidence knowledge tends to emerge in middle-to-lower layers.