NLP Systems That Can't Tell Use from Mention Censor Counterspeech, but Teaching the Distinction Helps
This addresses a critical issue for online content moderation by preventing the misclassification of non-harmful counterspeech as harmful, though it is incremental as it builds on existing prompting techniques.
The paper tackled the problem of language models failing to distinguish between the use and mention of words, which leads to censorship of counterspeech in misinformation and hate speech detection tasks. They introduced prompting mitigations that teach this distinction, reducing errors by showing concrete improvements in model performance.
The use of words to convey speaker's intent is traditionally distinguished from the `mention' of words for quoting what someone said, or pointing out properties of a word. Here we show that computationally modeling this use-mention distinction is crucial for dealing with counterspeech online. Counterspeech that refutes problematic content often mentions harmful language but is not harmful itself (e.g., calling a vaccine dangerous is not the same as expressing disapproval of someone for calling vaccines dangerous). We show that even recent language models fail at distinguishing use from mention, and that this failure propagates to two key downstream tasks: misinformation and hate speech detection, resulting in censorship of counterspeech. We introduce prompting mitigations that teach the use-mention distinction, and show they reduce these errors. Our work highlights the importance of the use-mention distinction for NLP and CSS and offers ways to address it.