Large Language Models for Toxic Language Detection in Low-Resource Balkan Languages
This addresses the problem of limited moderation tools for toxic content in underserved Balkan language communities, though it is incremental as it applies existing methods to new data with prompt design improvements.
The study tackled toxic language detection in low-resource Balkan languages (Serbian, Croatian, Bosnian) by evaluating four large language models on a manually labeled dataset of 4,500 YouTube/TikTok comments, finding that adding context snippets improved recall by 0.12 on average and F1 score by up to 0.10, with Gemini achieving the best balance at F1=0.82 and accuracy=0.82.
Online toxic language causes real harm, especially in regions with limited moderation tools. In this study, we evaluate how large language models handle toxic comments in Serbian, Croatian, and Bosnian, languages with limited labeled data. We built and manually labeled a dataset of 4,500 YouTube and TikTok comments drawn from videos across diverse categories, including music, politics, sports, modeling, influencer content, discussions of sexism, and general topics. Four models (GPT-3.5 Turbo, GPT-4.1, Gemini 1.5 Pro, and Claude 3 Opus) were tested in two modes: zero-shot and context-augmented. We measured precision, recall, F1 score, accuracy and false positive rates. Including a short context snippet raised recall by about 0.12 on average and improved F1 score by up to 0.10, though it sometimes increased false positives. The best balance came from Gemini in context-augmented mode, reaching an F1 score of 0.82 and accuracy of 0.82, while zero-shot GPT-4.1 led on precision and had the lowest false alarms. We show how adding minimal context can improve toxic language detection in low-resource settings and suggest practical strategies such as improved prompt design and threshold calibration. These results show that prompt design alone can yield meaningful gains in toxicity detection for underserved Balkan language communities.