KoDialogBench: Evaluating Conversational Understanding of Language Models with Korean Dialogue Benchmark
This work addresses the problem of evaluating conversational understanding in Korean for language model developers and users, though it is incremental as it adapts existing benchmarking approaches to a specific language.
The authors tackled the lack of comprehensive evaluation for language models' conversational proficiency in low-resource languages like Korean by introducing KoDialogBench, a benchmark for assessing Korean dialogue capabilities, and found significant room for improvement in models' conversation skills.
As language models are often deployed as chatbot assistants, it becomes a virtue for models to engage in conversations in a user's first language. While these models are trained on a wide range of languages, a comprehensive evaluation of their proficiency in low-resource languages such as Korean has been lacking. In this work, we introduce KoDialogBench, a benchmark designed to assess language models' conversational capabilities in Korean. To this end, we collect native Korean dialogues on daily topics from public sources, or translate dialogues from other languages. We then structure these conversations into diverse test datasets, spanning from dialogue comprehension to response selection tasks. Leveraging the proposed benchmark, we conduct extensive evaluations and analyses of various language models to measure a foundational understanding of Korean dialogues. Experimental results indicate that there exists significant room for improvement in models' conversation skills. Furthermore, our in-depth comparisons across different language models highlight the effectiveness of recent training techniques in enhancing conversational proficiency. We anticipate that KoDialogBench will promote the progress towards conversation-aware Korean language models.