Ex Machina: Personal Attacks Seen at Scale
This addresses the challenge of understanding personal attacks for online platforms like Wikipedia, revealing that attacks are not primarily from a few malicious users or anonymous contributions.
The paper tackled the problem of measuring personal attacks in online platforms at scale by developing a method combining crowdsourcing and machine learning, applying it to English Wikipedia to generate a corpus of over 100k human-labeled and 63M machine-labeled comments with a classifier as good as three crowd-workers.
The damage personal attacks cause to online discourse motivates many platforms to try to curb the phenomenon. However, understanding the prevalence and impact of personal attacks in online platforms at scale remains surprisingly difficult. The contribution of this paper is to develop and illustrate a method that combines crowdsourcing and machine learning to analyze personal attacks at scale. We show an evaluation method for a classifier in terms of the aggregated number of crowd-workers it can approximate. We apply our methodology to English Wikipedia, generating a corpus of over 100k high quality human-labeled comments and 63M machine-labeled ones from a classifier that is as good as the aggregate of 3 crowd-workers, as measured by the area under the ROC curve and Spearman correlation. Using this corpus of machine-labeled scores, our methodology allows us to explore some of the open questions about the nature of online personal attacks. This reveals that the majority of personal attacks on Wikipedia are not the result of a few malicious users, nor primarily the consequence of allowing anonymous contributions from unregistered users.