From Memorization to Creation: Evaluating the Cognitive Depth of LLM-Generated Educational Questions
For educators and AI developers, this study provides benchmarks and insights into designing prompts that enable LLMs to generate cognitively deeper educational content, though the improvements are incremental.
This work evaluates six LLMs on generating educational questions that require higher-order thinking, using a Bloom's Taxonomy framework. Key results include a 24.45% reduction in question repetitiveness and an 11.53% increase in higher-order cognitive outputs through fine-grained prompting.
While LLMs show promise in automating educational content creation, their ability to generate questions that stimulate higher-order thinking remains understudied. This work evaluates six widely-used LLMs through a Bloom's Taxonomy lens, focusing on their capacity to transcend rote memorization and achieve cognitive leaps. Using a hybrid human--AI evaluation protocol, we generate and analyze 20{,}700 questions across computer science, K--12 math, and social-science domains. Key contributions include: (1) a fine-grained prompting strategy that reduces question repetitiveness by 24.45\% for Qwen2.5-7B-Instruct, and increases the proportion of higher-order cognitive level outputs by 11.53\% for InternLM3-8B-Instruct; (2) quantitative metrics for cognitive shift intensity (CogShift) and category drift, revealing InternLM3's superior performance in multi-level transitions; (3) an interpretability analysis revealing metric-level correlations that enhance the transparency of Chain-of-Thought prompting. Our findings highlight the importance of cognitive-aware prompt design and provide benchmarks for deploying LLMs in personalized learning systems.