Routing Sensitivity Without Controllability: A Diagnostic Study of Fairness in MoE Language Models
For researchers and practitioners aiming to mitigate stereotypes in MoE models, the paper reveals fundamental architectural constraints that limit fairness control, informing future model design.
The paper diagnoses that Mixture-of-Experts language models are sensitive to demographic content at the routing level, but this sensitivity cannot be exploited for fairness control due to structural limitations. Experiments show that routing-level interventions either fail, are non-robust, or incur significant utility costs (e.g., -4.4%p CrowS-Pairs at -6.3%p TQA), and preference shifts do not transfer to generation.
Mixture-of-Experts (MoE) language models are universally sensitive to demographic content at the routing level, yet exploiting this sensitivity for fairness control is structurally limited. We introduce Fairness-Aware Routing Equilibrium (FARE), a diagnostic framework designed to probe the limits of routing-level stereotype intervention across diverse MoE architectures. FARE reveals that routing-level preference shifts are either unachievable (Mixtral, Qwen1.5, Qwen3), statistically non-robust (DeepSeekMoE), or accompanied by substantial utility cost (OLMoE, -4.4%p CrowS-Pairs at -6.3%p TQA). Critically, even where log-likelihood preference shifts are robust, they do not transfer to decoded generation: expanded evaluations on both non-null models yield null results across all generation metrics. Group-level expert masking reveals why: bias and core knowledge are deeply entangled within expert groups. These findings indicate that routing sensitivity is necessary but insufficient for stereotype control, and identify specific architectural conditions that can inform the design of more controllable future MoE systems.