Understanding LLM Development Through Longitudinal Study: Insights from the Open Ko-LLM Leaderboard
It provides a more comprehensive analysis for researchers and developers in Korean NLP, though it is incremental by extending prior short-term studies.
The paper conducted an 11-month longitudinal study of 1,769 models on the Open Ko-LLM Leaderboard to understand the progression of Korean LLM development, revealing insights into challenges, model size impacts, and ranking shifts over time.
This paper conducts a longitudinal study over eleven months to address the limitations of prior research on the Open Ko-LLM Leaderboard, which have relied on empirical studies with restricted observation periods of only five months. By extending the analysis duration, we aim to provide a more comprehensive understanding of the progression in developing Korean large language models (LLMs). Our study is guided by three primary research questions: (1) What are the specific challenges in improving LLM performance across diverse tasks on the Open Ko-LLM Leaderboard over time? (2) How does model size impact task performance correlations across various benchmarks? (3) How have the patterns in leaderboard rankings shifted over time on the Open Ko-LLM Leaderboard?. By analyzing 1,769 models over this period, our research offers a comprehensive examination of the ongoing advancements in LLMs and the evolving nature of evaluation frameworks.