Pragmatic Competence Evaluation of Large Language Models for the Korean Language
This addresses the gap in evaluating LLMs' pragmatic competence for Korean language users, though it is incremental as it applies existing methods to a new language domain.
The study evaluated how well large language models understand context-dependent expressions in Korean, finding that GPT-4 scored 81.11 in multiple-choice questions and 85.69 in open-ended questions, with HyperCLOVA X close behind.
Benchmarks play a significant role in the current evaluation of Large Language Models (LLMs), yet they often overlook the models' abilities to capture the nuances of human language, primarily focusing on evaluating embedded knowledge and technical skills. To address this gap, our study evaluates how well LLMs understand context-dependent expressions from a pragmatic standpoint, specifically in Korean. We use both Multiple-Choice Questions (MCQs) for automatic evaluation and Open-Ended Questions (OEQs) assessed by human experts. Our results show that GPT-4 leads with scores of 81.11 in MCQs and 85.69 in OEQs, closely followed by HyperCLOVA X. Additionally, while few-shot learning generally improves performance, Chain-of-Thought (CoT) prompting tends to encourage literal interpretations, which may limit effective pragmatic inference. Our findings highlight the need for LLMs to better understand and generate language that reflects human communicative norms.