Understanding and Analyzing Inappropriately Targeting Language in Online Discourse: A Comparative Annotation Study
This work addresses content moderation for online safety and inclusivity, but it is incremental as it builds on existing annotation methods with a comparative analysis.
The paper tackled the problem of detecting inappropriately targeting language in online discourse by comparing annotations from human experts, crowd workers, and ChatGPT on Reddit data, finding that contextual factors are crucial and revealing new categories like social belief and body image.
This paper introduces a method for detecting inappropriately targeting language in online conversations by integrating crowd and expert annotations with ChatGPT. We focus on English conversation threads from Reddit, examining comments that target individuals or groups. Our approach involves a comprehensive annotation framework that labels a diverse data set for various target categories and specific target words within the conversational context. We perform a comparative analysis of annotations from human experts, crowd annotators, and ChatGPT, revealing strengths and limitations of each method in recognizing both explicit hate speech and subtler discriminatory language. Our findings highlight the significant role of contextual factors in identifying hate speech and uncover new categories of targeting, such as social belief and body image. We also address the challenges and subjective judgments involved in annotation and the limitations of ChatGPT in grasping nuanced language. This study provides insights for improving automated content moderation strategies to enhance online safety and inclusivity.