Performance of Recent Large Language Models for a Low-Resourced Language
This work addresses the problem of poor LLM performance for low-resourced languages like Sinhala, which is incremental as it applies existing methods to new data.
The study evaluated four recent large language models on their performance for the low-resourced Sinhala language, finding that Claude and GPT 4o performed well out-of-the-box, while Llama and Mistral showed poor performance but potential for improvement with fine-tuning.
Large Language Models (LLMs) have shown significant advances in the past year. In addition to new versions of GPT and Llama, several other LLMs have been introduced recently. Some of these are open models available for download and modification. Although multilingual large language models have been available for some time, their performance on low-resourced languages such as Sinhala has been poor. We evaluated four recent LLMs on their performance directly in the Sinhala language, and by translation to and from English. We also evaluated their fine-tunability with a small amount of fine-tuning data. Claude and GPT 4o perform well out-of-the-box and do significantly better than previous versions. Llama and Mistral perform poorly but show some promise of improvement with fine tuning.