Challenges in Adapting Multilingual LLMs to Low-Resource Languages using LoRA PEFT Tuning
This addresses challenges in adapting LLMs for low-resource languages like Marathi, though it is incremental as it applies existing methods to new data.
The study investigated adapting multilingual Gemma models to Marathi using LoRA PEFT tuning with a translated Alpaca dataset, finding that fine-tuning often improves target language generation but reduces reasoning abilities, with evaluation metrics showing performance declines while manual assessments suggest better performance.
Large Language Models (LLMs) have demonstrated remarkable multilingual capabilities, yet challenges persist in adapting these models for low-resource languages. In this study, we investigate the effects of Low-Rank Adaptation (LoRA) Parameter-Efficient Fine-Tuning (PEFT) on multilingual Gemma models for Marathi, a language with limited resources. Using a translated Alpaca dataset with 52,000 instruction-response pairs, our findings reveal that while evaluation metrics often show a performance decline post-fine-tuning, manual assessments frequently suggest that the fine-tuned models outperform their original counterparts. The observations indicate improvements in target language generation capabilities but a reduction in reasoning abilities following language adaptation. These results underscore the need for improved evaluation methodologies and the creation of high-quality native datasets to accurately assess language-specific model performance in low-resource settings.