APEACH: Attacking Pejorative Expressions with Analysis on Crowd-Generated Hate Speech Evaluation Datasets
This addresses the issue of overestimated performance in hate speech detection for low-data languages like Korean, though it is incremental as it builds on existing crowd-sourcing approaches.
The paper tackles the problem of limited generalization in hate speech detection datasets by proposing APEACH, a method that collects crowd-generated hate speech examples with minimal post-labeling in Korean, resulting in datasets less sensitive to lexical overlaps and better model performance measurement.
In hate speech detection, developing training and evaluation datasets across various domains is the critical issue. Whereas, major approaches crawl social media texts and hire crowd-workers to annotate the data. Following this convention often restricts the scope of pejorative expressions to a single domain lacking generalization. Sometimes domain overlap between training corpus and evaluation set overestimate the prediction performance when pretraining language models on low-data language. To alleviate these problems in Korean, we propose APEACH that asks unspecified users to generate hate speech examples followed by minimal post-labeling. We find that APEACH can collect useful datasets that are less sensitive to the lexical overlaps between the pretraining corpus and the evaluation set, thereby properly measuring the model performance.