HateXScore: A Metric Suite for Evaluating Reasoning Quality in Hate Speech Explanations
This addresses the need for better evaluation in hate speech detection for content moderation, though it is incremental as it builds on existing interpretability metrics.
The paper tackles the problem of evaluating reasoning quality in hate speech explanations by introducing HateXScore, a metric suite that assesses four components, and shows strong human agreement in validation.
Hateful speech detection is a key component of content moderation, yet current evaluation frameworks rarely assess why a text is deemed hateful. We introduce \textsf{HateXScore}, a four-component metric suite designed to evaluate the reasoning quality of model explanations. It assesses (i) conclusion explicitness, (ii) faithfulness and causal grounding of quoted spans, (iii) protected group identification (policy-configurable), and (iv) logical consistency among these elements. Evaluated on six diverse hate speech datasets, \textsf{HateXScore} is intended as a diagnostic complement to reveal interpretability failures and annotation inconsistencies that are invisible to standard metrics like Accuracy or F1. Moreover, human evaluation shows strong agreement with \textsf{HateXScore}, validating it as a practical tool for trustworthy and transparent moderation. \textcolor{red}{Disclaimer: This paper contains sensitive content that may be disturbing to some readers.}