SinLlama -- A Large Language Model for Sinhala
This addresses the problem of low-resource language support in LLMs for Sinhala speakers, though it is incremental as it builds on an existing model.
The researchers tackled the lack of open-source LLMs for Sinhala by extending Llama-3-8B with Sinhala vocabulary and continual pre-training on a 10 million Sinhala corpus, resulting in SinLlama, which outperformed base and instruct variants of Llama-3-8B in text classification tasks by a significant margin.
Low-resource languages such as Sinhala are often overlooked by open-source Large Language Models (LLMs). In this research, we extend an existing multilingual LLM (Llama-3-8B) to better serve Sinhala. We enhance the LLM tokenizer with Sinhala specific vocabulary and perform continual pre-training on a cleaned 10 million Sinhala corpus, resulting in the SinLlama model. This is the very first decoder-based open-source LLM with explicit Sinhala support. When SinLlama was instruction fine-tuned for three text classification tasks, it outperformed base and instruct variants of Llama-3-8B by a significant margin.