Fine-Tuning Language Models to Know What They Know
This addresses the problem of enhancing metacognitive awareness in language models for AI researchers, though it appears incremental as it builds on existing methods for aligning internal knowledge with explicit behaviors.
The study tackled the problem of measuring and improving metacognitive ability in language models, specifically their awareness of their own knowledge, by introducing a framework to measure metacognitive ability and the Evolution Strategy for Metacognitive Alignment (ESMA) method, which demonstrated robust generalization across diverse untrained settings and attributed improvements to sparse parameter modifications.
Metacognition is a critical component of intelligence, specifically regarding the awareness of one's own knowledge. While humans rely on shared internal memory for both answering questions and reporting their knowledge state, this dependency in LLMs remains underexplored. This study proposes a framework to measure metacognitive ability $d_{\rm{type2}}'$ using a dual-prompt method, followed by the introduction of Evolution Strategy for Metacognitive Alignment (ESMA) to bind a model's internal knowledge to its explicit behaviors. ESMA demonstrates robust generalization across diverse untrained settings, indicating a enhancement in the model's ability to reference its own knowledge. Furthermore, parameter analysis attributes these improvements to a sparse set of significant modifications.