Dr.Academy: A Benchmark for Evaluating Questioning Capability in Education for Large Language Models
This work addresses the need to assess LLMs as educators rather than learners, which is important for advancing automated and personalized learning tools, though it is incremental as it builds on existing evaluation methods by focusing on a specific teaching skill.
The paper tackles the problem of evaluating large language models' (LLMs) teaching capabilities, specifically their questioning ability in education, by introducing a benchmark that assesses generated educational questions across general, monodisciplinary, and interdisciplinary domains, with results showing GPT-4 excels in general, humanities, and science courses while Claude2 performs better in interdisciplinary contexts, and automatic scores align with human evaluations.
Teachers are important to imparting knowledge and guiding learners, and the role of large language models (LLMs) as potential educators is emerging as an important area of study. Recognizing LLMs' capability to generate educational content can lead to advances in automated and personalized learning. While LLMs have been tested for their comprehension and problem-solving skills, their capability in teaching remains largely unexplored. In teaching, questioning is a key skill that guides students to analyze, evaluate, and synthesize core concepts and principles. Therefore, our research introduces a benchmark to evaluate the questioning capability in education as a teacher of LLMs through evaluating their generated educational questions, utilizing Anderson and Krathwohl's taxonomy across general, monodisciplinary, and interdisciplinary domains. We shift the focus from LLMs as learners to LLMs as educators, assessing their teaching capability through guiding them to generate questions. We apply four metrics, including relevance, coverage, representativeness, and consistency, to evaluate the educational quality of LLMs' outputs. Our results indicate that GPT-4 demonstrates significant potential in teaching general, humanities, and science courses; Claude2 appears more apt as an interdisciplinary teacher. Furthermore, the automatic scores align with human perspectives.