When Debiasing Backfires: Counterintuitive Side Effects of Preprocessing-Based Stereotype Mitigation
For NLP practitioners and fairness researchers, this work reveals that common debiasing techniques can backfire, highlighting the need for side-effect-aware evaluation and mitigation.
Preprocessing-based debiasing methods in NLP reduce stereotypes for targeted groups but often cause unintended side effects, increasing stereotyping or counter-stereotyping for other demographics, including across unrelated categories, across model families and data scales.
Preprocessing-based methods for stereotype mitigation, such as pre-/post-training on debiased corpora, are widely used in NLP. While these approaches reduce measurable stereotypes for targeted groups, we find they often induce unintended shifts-side effects, where stereotyping or counter-stereotyping can increase relative to neutral baselines for other demographics, including across unrelated demographic categories. We demonstrate these side effects across two model families (encoder-only and decoder-only), multiple preprocessing strategies (removing stereotypical sentences, removing group mentions, and swapping group references), and both pre- and post-training at different data scales on Wikipedia. Standard benchmarks frequently miss these shifts. Using attention-rollout analysis, we observe that such side effects are not accompanied by large changes in attention flow, complicating mechanistic explanations. We discuss implications for evaluation, provide actionable diagnostics, and argue for side-effect-aware, transparent mitigation practices.