Vision-Language Models Suppress Female Representations Under Ambiguous Input
This study identifies a critical issue of gender bias in VLM outputs for ambiguous inputs, impacting fairness and representation in AI systems.
This paper investigates how vision-language models (VLMs) represent gender in ambiguous images, finding that they default to male representations even for female-stereotyped occupations. The authors introduce LALS, a zero-shot metric, to show that internal model representations often encode female associations but output male, due to an asymmetric filter that suppresses female signals.
Alignment teaches vision-language models (VLMs) to avoid expressing demographic biases, and when gender is clearly visible they largely succeed. Far less is known about ambiguous inputs (a worker in full gear, a figure seen from behind) cases common in practice yet rarely studied. We find that minimal prompting pressure exposes occupation-gender defaults when prompting ambiguous input images, with models collapsing to male even for strongly female-stereotyped occupations. But do these outputs reflect what models actually encode internally? We introduce LALS (Latent Association Leaning Score), a zero-shot metric that projects visual-token activations into the model's text-embedding space to measure concept associations per token and layer. Across 15 occupations, over 800 gender-ambiguous images, and four VLMs, internal representations and outputs are systematically decoupled: models often encode a female association internally yet output male. Layer-wise analysis reveals an asymmetric filter -- male signal amplifies end-to-end while female signal peaks mid-network and is suppressed before generation -- and a color ablation shows that culturally loaded visual cues such as clothing color further modulate these internal associations.