When a Language Question Is at Stake. A Revisited Approach to Label Sensitive Content
This work addresses the challenge of sensitive content annotation for under-resourced languages, offering incremental improvements to reduce annotator burden in a domain-specific context.
The paper tackles the problem of creating high-quality datasets for offensive language and disinformation detection in under-resourced languages like Ukrainian, focusing on tweets from the Russian-Ukrainian war, by revisiting a pseudo-labeling approach to avoid annotator harm; it provides statistical analysis, model evaluations, and guidelines for extending datasets without human annotation.
Many under-resourced languages require high-quality datasets for specific tasks such as offensive language detection, disinformation, or misinformation identification. However, the intricacies of the content may have a detrimental effect on the annotators. The article aims to revisit an approach of pseudo-labeling sensitive data on the example of Ukrainian tweets covering the Russian-Ukrainian war. Nowadays, this acute topic is in the spotlight of various language manipulations that cause numerous disinformation and profanity on social media platforms. The conducted experiment highlights three main stages of data annotation and underlines the main obstacles during machine annotation. Ultimately, we provide a fundamental statistical analysis of the obtained data, evaluation of models used for pseudo-labelling, and set further guidelines on how the scientists can leverage the corpus to execute more advanced research and extend the existing data samples without annotators' engagement.