LuxIT: A Luxembourgish Instruction Tuning Dataset from Monolingual Seed Data
This addresses the problem of limited LLM effectiveness for low-resource languages like Luxembourgish, though it is incremental as it applies an existing methodology to a new linguistic context.
The authors tackled the lack of high-quality instruction tuning data for low-resource languages by creating LuxIT, a monolingual dataset for Luxembourgish synthesized from native texts using DeepSeek-R1-0528, but fine-tuning smaller LLMs on it yielded mixed results with varying performance on language proficiency exams.
The effectiveness of instruction-tuned Large Language Models (LLMs) is often limited in low-resource linguistic settings due to a lack of high-quality training data. We introduce LuxIT, a novel, monolingual instruction tuning dataset for Luxembourgish developed to mitigate this challenge. We synthesize the dataset from a corpus of native Luxembourgish texts, utilizing DeepSeek-R1-0528, chosen for its shown proficiency in Luxembourgish. Following generation, we apply a quality assurance process, employing an LLM-as-a-judge approach. To investigate the practical utility of the dataset, we fine-tune several smaller-scale LLMs on LuxIT. Subsequent benchmarking against their base models on Luxembourgish language proficiency examinations, however, yields mixed results, with performance varying significantly across different models. LuxIT represents a critical contribution to Luxembourgish natural language processing and offers a replicable monolingual methodology, though our findings highlight the need for further research to optimize its application.