Alignment Reduces Expressed but Not Encoded Gender Bias: A Unified Framework and Study
This work addresses the limitation of current bias mitigation efforts for LLMs by showing that alignment may not generalize to realistic scenarios, which is important for developers and users concerned with ethical AI.
The study tackled the problem of gender bias in Large Language Models by proposing a unified framework to analyze both encoded and expressed bias, finding that alignment reduces expressed bias but not encoded bias, which can be reactivated under adversarial prompting.
During training, Large Language Models (LLMs) learn social regularities that can lead to gender bias in downstream applications. Most mitigation efforts focus on reducing bias in generated outputs, typically evaluated on structured benchmarks, which raises two concerns: output-level evaluation does not reveal whether alignment modifies the model's underlying representations, and structured benchmarks may not reflect realistic usage scenarios. We propose a unified framework to jointly analyze intrinsic and extrinsic gender bias in LLMs using identical neutral prompts, enabling direct comparison between gender-related information encoded in internal representations and bias expressed in generated outputs. Contrary to prior work reporting weak or inconsistent correlations, we find a consistent association between latent gender information and expressed bias when measured under the unified protocol. We further examine the effect of alignment through supervised fine-tuning aimed at reducing gender bias. Our results suggest that while the latter indeed reduces expressed bias, measurable gender-related associations are still present in internal representations, and can be reactivated under adversarial prompting. Finally, we consider two realistic settings and show that debiasing effects observed on structured benchmarks do not necessarily generalize, e.g., to the case of story generation.