AlbanianLLMSafety: A Safety Evaluation Dataset for Large Language Models in Albanian
This dataset provides a benchmark for safety evaluation of LLMs in Albanian, a low-resource language with 7.5 million speakers, enabling safer model development for underserved communities.
The authors created the first safety evaluation dataset for LLMs in Albanian, containing 2,951 prompts across 11 categories, addressing the gap in safety resources for low-resource languages.
Safety evaluation of Large Language Models (LLMs) has largely focused on high-resource languages, leaving low-resource languages critically underserved. We present AlbanianLLMSafety, the first publicly available safety evaluation dataset for LLMs in Albanian, a linguistically distinct low-resource language with approximately 7.5 million speakers across Albania, Kosovo, North Macedonia, and the diaspora. The dataset contains 2,951 prompts spanning 11 safety categories, including self-harm, violence, racist content, child exploitation, and radicalization, with an average of 268 prompts per category. Each prompt is provided in Albanian with an English reference translation and a detailed category label. This resource addresses a significant gap in safety evaluation infrastruc-ture for low-resource languages and provides an essential benchmark for developing safer, more inclusive LLMs. The dataset will be provided upon request to support safety evaluation, fine-tuning, red-teaming, and guardrail development for Albanian-speaking communities.