Generalizing Hate Speech Detection Using Multi-Task Learning: A Case Study of Political Public Figures
This work addresses the challenge of generalizing hate speech detection for online content moderation, though it is incremental as it builds on existing multi-task learning methods.
The paper tackles the problem of poor generalization in hate speech detection across different datasets by proposing a Multi-task Learning pipeline that trains on multiple datasets simultaneously, achieving consistent outperformance over existing work and substantial improvements on unseen datasets. It also introduces a novel dataset, PubFigs, with over 305,000 tweets from American political figures, finding that abusive content primarily originates from right-leaning figures and relates to topics like Islam and ethnicity.
Automatic identification of hateful and abusive content is vital in combating the spread of harmful online content and its damaging effects. Most existing works evaluate models by examining the generalization error on train-test splits on hate speech datasets. These datasets often differ in their definitions and labeling criteria, leading to poor generalization performance when predicting across new domains and datasets. This work proposes a new Multi-task Learning (MTL) pipeline that trains simultaneously across multiple hate speech datasets to construct a more encompassing classification model. Using a dataset-level leave-one-out evaluation (designating a dataset for testing and jointly training on all others), we trial the MTL detection on new, previously unseen datasets. Our results consistently outperform a large sample of existing work. We show strong results when examining the generalization error in train-test splits and substantial improvements when predicting on previously unseen datasets. Furthermore, we assemble a novel dataset, dubbed PubFigs, focusing on the problematic speech of American Public Political Figures. We crowdsource-label using Amazon MTurk more than $20,000$ tweets and machine-label problematic speech in all the $305,235$ tweets in PubFigs. We find that the abusive and hate tweeting mainly originates from right-leaning figures and relates to six topics, including Islam, women, ethnicity, and immigrants. We show that MTL builds embeddings that can simultaneously separate abusive from hate speech, and identify its topics.