Türkçe Dil Modellerinin Performans Karşılaştırması Performance Comparison of Turkish Language Models
It addresses the lack of comprehensive performance comparisons for Turkish language models, which is important for users preferring open-source options due to cost or privacy concerns, though it is incremental as it applies existing methods to a new language.
This study compared the performance of seven language models on Turkish tasks, finding that continuing pretraining before fine-tuning with instructional datasets improves question-answering adaptation for multilingual models, and that in-context learning performance is not strongly related to question-answering performance.
The developments that language models have provided in fulfilling almost all kinds of tasks have attracted the attention of not only researchers but also the society and have enabled them to become products. There are commercially successful language models available. However, users may prefer open-source language models due to cost, data privacy, or regulations. Yet, despite the increasing number of these models, there is no comprehensive comparison of their performance for Turkish. This study aims to fill this gap in the literature. A comparison is made among seven selected language models based on their contextual learning and question-answering abilities. Turkish datasets for contextual learning and question-answering were prepared, and both automatic and human evaluations were conducted. The results show that for question-answering, continuing pretraining before fine-tuning with instructional datasets is more successful in adapting multilingual models to Turkish and that in-context learning performances do not much related to question-answering performances.