Text Speaks Louder than Vision: ASCII Art Reveals Textual Biases in Vision-Language Models
This work uncovers fundamental flaws in multimodal integration for VLMs, with implications for content moderation systems vulnerable to adversarial examples, though it is incremental in highlighting a known bottleneck.
The study investigated how vision-language models handle conflicting signals in ASCII art, revealing a strong text-priority bias where models consistently favor textual over visual information, with visual recognition declining by up to 40% as semantic complexity increased.
Vision-language models (VLMs) have advanced rapidly in processing multimodal information, but their ability to reconcile conflicting signals across modalities remains underexplored. This work investigates how VLMs process ASCII art, a unique medium where textual elements collectively form visual patterns, potentially creating semantic-visual conflicts. We introduce a novel evaluation framework that systematically challenges five state-of-the-art models (including GPT-4o, Claude, and Gemini) using adversarial ASCII art, where character-level semantics deliberately contradict global visual patterns. Our experiments reveal a strong text-priority bias: VLMs consistently prioritize textual information over visual patterns, with visual recognition ability declining dramatically as semantic complexity increases. Various mitigation attempts through visual parameter tuning and prompt engineering yielded only modest improvements, suggesting that this limitation requires architectural-level solutions. These findings uncover fundamental flaws in how current VLMs integrate multimodal information, providing important guidance for future model development while highlighting significant implications for content moderation systems vulnerable to adversarial examples.