VISTA-Bench: Do Vision-Language Models Really Understand Visualized Text as Well as Pure Text?
This addresses a critical limitation for real-world applications of VLMs, where text often appears in images, but the work is incremental as it builds on existing benchmarking efforts.
The paper tackles the problem of whether vision-language models (VLMs) understand visualized text in images as well as pure text, by introducing VISTA-Bench, a benchmark that reveals a pronounced modality gap where models degrade substantially on visualized-text queries compared to pure-text ones, with performance dropping by up to 30% in some cases.
Vision-Language Models (VLMs) have achieved impressive performance in cross-modal understanding across textual and visual inputs, yet existing benchmarks predominantly focus on pure-text queries. In real-world scenarios, language also frequently appears as visualized text embedded in images, raising the question of whether current VLMs handle such input requests comparably. We introduce VISTA-Bench, a systematic benchmark from multimodal perception, reasoning, to unimodal understanding domains. It evaluates visualized text understanding by contrasting pure-text and visualized-text questions under controlled rendering conditions. Extensive evaluation of over 20 representative VLMs reveals a pronounced modality gap: models that perform well on pure-text queries often degrade substantially when equivalent semantic content is presented as visualized text. This gap is further amplified by increased perceptual difficulty, highlighting sensitivity to rendering variations despite unchanged semantics. Overall, VISTA-Bench provides a principled evaluation framework to diagnose this limitation and to guide progress toward more unified language representations across tokenized text and pixels. The source dataset is available at https://github.com/QingAnLiu/VISTA-Bench.