Centering the Margins: Outlier-Based Identification of Harmed Populations in Toxicity Detection
This work addresses the issue of obscured harms against marginalized and intersectional groups in AI toxicity detection, offering a novel approach to surface greater harms compared to traditional demographic breakdowns.
The paper tackled the problem of identifying harmed populations in toxicity detection by using outlier detection to find demographic and text outliers, revealing that model performance is significantly worse for outliers with mean squared error up to 70.4% higher for demographic outliers and 68.4% higher for text outliers.
The impact of AI models on marginalized communities has traditionally been measured by identifying performance differences between specified demographic subgroups. Though this approach aims to center vulnerable groups, it risks obscuring patterns of harm faced by intersectional subgroups or shared across multiple groups. To address this, we draw on theories of marginalization from disability studies and related disciplines, which state that people farther from the norm face greater adversity, to consider the "margins" in the domain of toxicity detection. We operationalize the "margins" of a dataset by employing outlier detection to identify text about people with demographic attributes distant from the "norm". We find that model performance is consistently worse for demographic outliers, with mean squared error (MSE) between outliers and non-outliers up to 70.4% worse across toxicity types. It is also worse for text outliers, with a MSE up to 68.4% higher for outliers than non-outliers. We also find text and demographic outliers to be particularly susceptible to errors in the classification of severe toxicity and identity attacks. Compared to analysis of disparities using traditional demographic breakdowns, we find that our outlier analysis frequently surfaces greater harms faced by a larger, more intersectional group, which suggests that outlier analysis is particularly beneficial for identifying harms against those groups.