Assessing Thai Dialect Performance in LLMs with Automatic Benchmarks and Human Evaluation
This work addresses the lack of benchmarks for local dialects in NLP, which is an incremental step for researchers and developers focusing on underrepresented languages.
The paper tackled the problem of evaluating large language models (LLMs) on underrepresented Thai local dialects, finding that performance declines significantly compared to standard Thai, with only proprietary models like GPT-4o and Gemini2 showing some fluency.
Large language models show promising results in various NLP tasks. Despite these successes, the robustness and consistency of LLMs in underrepresented languages remain largely unexplored, especially concerning local dialects. Existing benchmarks also focus on main dialects, neglecting LLMs' ability on local dialect texts. In this paper, we introduce a Thai local dialect benchmark covering Northern (Lanna), Northeastern (Isan), and Southern (Dambro) Thai, evaluating LLMs on five NLP tasks: summarization, question answering, translation, conversation, and food-related tasks. Furthermore, we propose a human evaluation guideline and metric for Thai local dialects to assess generation fluency and dialect-specific accuracy. Results show that LLM performance declines significantly in local Thai dialects compared to standard Thai, with only proprietary models like GPT-4o and Gemini2 demonstrating some fluency