Large Language Models Have Intrinsic Meta-Cognition, but Need a Good Lens
This work addresses the reliability of LLMs for users in AI and reasoning tasks, but it is incremental as it builds on existing self-evaluation methods.
The paper tackles the problem of evaluating and improving the meta-cognitive abilities of Large Language Models (LLMs), such as self-awareness of errors in reasoning steps, by proposing AutoMeco for benchmarking and MIRA to enhance evaluation lenses, showing improved assessment on three mathematical reasoning datasets and three LLMs.
Previous research has primarily focused on the cognitive error detection capabilities of Large Language Models (LLMs), often prompting them to analyze mistakes in reasoning chains. However, few studies have examined the meta-cognitive abilities of LLMs (e.g., their self-awareness of step errors), which are crucial for their reliability. While studies on LLM self-evaluation present some measures, such as perplexity, which can reflect the answer correctness and be viewed as the lens of meta-cognition, they lack step-level analysis and adaptation. This paper studies the evaluation of LLM meta-cognition using the current lenses and how to improve these lenses. Specifically, we propose AutoMeco, an Automated Meta-cognition Evaluation framework for benchmarking the existing lenses. Furthermore, a training-free Markovian Intrinsic Reward Adjustment strategy, MIRA, is proposed to boost current meta-cognition lenses. Experimental results on three mathematical reasoning datasets and three LLMs show the reasonableness of AutoMeco by comparing it with Best-of-N verification. Moreover, the meta-cognition ability of LLMs can be better evaluated using MIRA.