KULTURE Bench: A Benchmark for Assessing Language Model in Korean Cultural Context
This addresses the issue of cultural bias in multilingual benchmarks for researchers and developers working on Korean language models, though it is incremental as it focuses on a specific domain.
The authors tackled the problem of evaluating language models in non-Western cultural contexts by introducing KULTURE Bench, a benchmark for Korean culture, and found that models still have significant room for improvement in understanding deeper aspects of Korean culture.
Large language models have exhibited significant enhancements in performance across various tasks. However, the complexity of their evaluation increases as these models generate more fluent and coherent content. Current multilingual benchmarks often use translated English versions, which may incorporate Western cultural biases that do not accurately assess other languages and cultures. To address this research gap, we introduce KULTURE Bench, an evaluation framework specifically designed for Korean culture that features datasets of cultural news, idioms, and poetry. It is designed to assess language models' cultural comprehension and reasoning capabilities at the word, sentence, and paragraph levels. Using the KULTURE Bench, we assessed the capabilities of models trained with different language corpora and analyzed the results comprehensively. The results show that there is still significant room for improvement in the models' understanding of texts related to the deeper aspects of Korean culture.