FC-CONAN: An Exhaustively Paired Dataset for Robust Evaluation of Retrieval Systems
This dataset addresses the need for robust evaluation in counterspeech research, though it is incremental as it builds on existing resources like CONAN.
The authors tackled the problem of evaluating hate speech counter-narrative retrieval systems by creating FC-CONAN, a dataset that exhaustively pairs 45 hate speech messages with 129 counter-narratives, resulting in hundreds of newly labeled positive pairs across four reliability partitions.
Hate speech (HS) is a critical issue in online discourse, and one promising strategy to counter it is through the use of counter-narratives (CNs). Datasets linking HS with CNs are essential for advancing counterspeech research. However, even flagship resources like CONAN (Chung et al., 2019) annotate only a sparse subset of all possible HS-CN pairs, limiting evaluation. We introduce FC-CONAN (Fully Connected CONAN), the first dataset created by exhaustively considering all combinations of 45 English HS messages and 129 CNs. A two-stage annotation process involving nine annotators and four validators produces four partitions-Diamond, Gold, Silver, and Bronze-that balance reliability and scale. None of the labeled pairs overlap with CONAN, uncovering hundreds of previously unlabelled positives. FC-CONAN enables more faithful evaluation of counterspeech retrieval systems and facilitates detailed error analysis. The dataset is publicly available.