From Ground Trust to Truth: Disparities in Offensive Language Judgments on Contemporary Korean Political Discourse
This work addresses the need for up-to-date evaluation in offensive language detection for researchers and practitioners, though it is incremental in nature.
The study tackled the problem of outdated datasets in offensive language detection by constructing a contemporary Korean political discourse dataset and using refined judgments to assess performance, finding that a single prompting method achieved comparable results to more resource-intensive approaches.
Although offensive language continually evolves over time, even recent studies using LLMs have predominantly relied on outdated datasets and rarely evaluated the generalization ability on unseen texts. In this study, we constructed a large-scale dataset of contemporary political discourse and employed three refined judgments in the absence of ground truth. Each judgment reflects a representative offensive language detection method and is carefully designed for optimal conditions. We identified distinct patterns for each judgment and demonstrated tendencies of label agreement using a leave-one-out strategy. By establishing pseudo-labels as ground trust for quantitative performance assessment, we observed that a strategically designed single prompting achieves comparable performance to more resource-intensive methods. This suggests a feasible approach applicable in real-world settings with inherent constraints.