Thunder-LLM: Efficiently Adapting LLMs to Korean with Minimal Resources
This addresses the issue of LLM underperformance in non-English languages like Korean for users needing cost-effective language adaptation, though it is incremental as it builds on existing methods for a specific domain.
The paper tackles the problem of adapting English-based large language models (LLMs) to Korean efficiently with minimal resources, resulting in Thunder-LLM models that achieve superior Korean performance compared to state-of-the-art models while using minimal data and computational resources.
Since state-of-the-art LLMs often underperform in languages other than English or Chinese, improving the capability of LLMs in new languages has become an essential task. Moreover, LLMs' entire end-to-end training process remains largely unknown to the public due to proprietary reasons, technical complexity, inconsistent documentation, and ethical considerations. The complete picture remains a closely guarded secret within the industry. This paper presents methods to adapt an existing English-based LLM to Korean in a low-budget scenario. We describe the entire end-to-end process: collecting Korean datasets, preprocessing the data, training the model, creating downstream benchmarks, and conducting evaluations. The evaluation results indicate that our method can effectively and cost-efficiently add new language capabilities to existing LLMs. Our new bilingual models, Thunder-LLM and Thunder-LLM-Ins, achieve superior Korean performance compared to state-of-the-art models while utilizing minimal data and computational resources. We share our comprehensive experience and make the code publicly available.