From Dogwhistles to Bullhorns: Unveiling Coded Rhetoric with Language Models
This work addresses the challenge of identifying harmful coded language for NLP and computational social science, providing resources for future research, but it is incremental as it builds on existing language model capabilities.
This paper tackles the problem of detecting dogwhistles, coded expressions with hidden meanings, by conducting the first large-scale computational investigation, curating a glossary of over 300 dogwhistles and analyzing their usage in U.S. political speeches, and finds that GPT-3's detection performance varies and such content evades toxicity detection.
Dogwhistles are coded expressions that simultaneously convey one meaning to a broad audience and a second one, often hateful or provocative, to a narrow in-group; they are deployed to evade both political repercussions and algorithmic content moderation. For example, in the sentence 'we need to end the cosmopolitan experiment,' the word 'cosmopolitan' likely means 'worldly' to many, but secretly means 'Jewish' to a select few. We present the first large-scale computational investigation of dogwhistles. We develop a typology of dogwhistles, curate the largest-to-date glossary of over 300 dogwhistles with rich contextual information and examples, and analyze their usage in historical U.S. politicians' speeches. We then assess whether a large language model (GPT-3) can identify dogwhistles and their meanings, and find that GPT-3's performance varies widely across types of dogwhistles and targeted groups. Finally, we show that harmful content containing dogwhistles avoids toxicity detection, highlighting online risks of such coded language. This work sheds light on the theoretical and applied importance of dogwhistles in both NLP and computational social science, and provides resources for future research in modeling dogwhistles and mitigating their online harms.