Building a Llama2-finetuned LLM for Odia Language Utilizing Domain Knowledge Instruction Set
This addresses the need for LLMs in low-resource Indic languages like Odia to provide AI technologies to citizens in multilingual regions such as India, though it is incremental as it applies existing fine-tuning methods to new data.
The paper tackles the problem of building LLMs for low-resource languages by generating a large Odia instruction set with domain knowledge and fine-tuning Llama2 for enhanced performance in Odia, aiming to support Indic languages in generative AI services.
Building LLMs for languages other than English is in great demand due to the unavailability and performance of multilingual LLMs, such as understanding the local context. The problem is critical for low-resource languages due to the need for instruction sets. In a multilingual country like India, there is a need for LLMs supporting Indic languages to provide generative AI and LLM-based technologies and services to its citizens. This paper presents our approach of i) generating a large Odia instruction set, including domain knowledge data suitable for LLM fine-tuning, and ii) building a Llama2-finetuned model tailored for enhanced performance in the Odia domain. The proposed work will help researchers build an instruction set and LLM, particularly for Indic languages. We will release the model and instruction set for the public for research and noncommercial purposes.