Probing the Difficulty Perception Mechanism of Large Language Models
This work addresses the problem of understanding LLMs' internal difficulty perception for researchers and practitioners, offering potential applications in automatic difficulty annotation to reduce human labeling costs, though it is incremental in nature.
The study investigated whether large language models (LLMs) implicitly encode problem difficulty in their internal representations, finding that difficulty levels of math problems can be linearly modeled from final-token representations with specific attention heads showing opposite activation patterns for simple and difficult problems.
Large language models (LLMs) are increasingly deployed on complex reasoning tasks, yet little is known about their ability to internally evaluate problem difficulty, which is an essential capability for adaptive reasoning and efficient resource allocation. In this work, we investigate whether LLMs implicitly encode problem difficulty in their internal representations. Using a linear probe on the final-token representations of LLMs, we demonstrate that the difficulty level of math problems can be linearly modeled. We further locate the specific attention heads of the final Transformer layer: these attention heads have opposite activation patterns for simple and difficult problems, thus achieving perception of difficulty. Our ablation experiments prove the accuracy of the location. Crucially, our experiments provide practical support for using LLMs as automatic difficulty annotators, potentially substantially reducing reliance on costly human labeling in benchmark construction and curriculum learning. We also uncover that there is a significant difference in entropy and difficulty perception at the token level. Our study reveals that difficulty perception in LLMs is not only present but also structurally organized, offering new theoretical insights and practical directions for future research. Our code is available at https://github.com/Aegis1863/Difficulty-Perception-of-LLMs.