Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activations
This addresses AI safety concerns by providing empirical evidence for metacognition in LLMs, which is crucial for understanding and mitigating risks like adversarial attacks, though it is incremental in building on existing neuroscience and in-context learning methods.
The paper tackles the problem of quantifying metacognitive abilities in large language models (LLMs) by introducing a neuroscience-inspired neurofeedback paradigm, demonstrating that LLMs can monitor and control only a small subset of their neural activations, with factors like in-context examples and activation interpretability affecting these abilities.
Large language models (LLMs) can sometimes report the strategies they actually use to solve tasks, yet at other times seem unable to recognize those strategies that govern their behavior. This suggests a limited degree of metacognition - the capacity to monitor one's own cognitive processes for subsequent reporting and self-control. Metacognition enhances LLMs' capabilities in solving complex tasks but also raises safety concerns, as models may obfuscate their internal processes to evade neural-activation-based oversight (e.g., safety detector). Given society's increased reliance on these models, it is critical that we understand their metacognitive abilities. To address this, we introduce a neuroscience-inspired neurofeedback paradigm that uses in-context learning to quantify metacognitive abilities of LLMs to report and control their activation patterns. We demonstrate that their abilities depend on several factors: the number of in-context examples provided, the semantic interpretability of the neural activation direction (to be reported/controlled), and the variance explained by that direction. These directions span a "metacognitive space" with dimensionality much lower than the model's neural space, suggesting LLMs can monitor only a small subset of their neural activations. Our paradigm provides empirical evidence to quantify metacognition in LLMs, with significant implications for AI safety (e.g., adversarial attack and defense).