VisTW: Benchmarking Vision-Language Models for Traditional Chinese in Taiwan
This addresses a critical gap for researchers and developers working with Traditional Chinese in regions like Taiwan and Hong Kong, though it is incremental as it extends existing benchmarking efforts to a new linguistic context.
The authors tackled the lack of evaluation benchmarks for Vision-Language Models in Traditional Chinese by creating VisTW, a comprehensive suite with multiple-choice and dialogue components, revealing significant performance differences and specific challenges in processing Traditional Chinese visual content.
In this paper, we propose a comprehensive evaluation benchmark for Visual Language Models (VLM) in Traditional Chinese. Our evaluation suite, the first of its kind, contains two complementary components: (1) VisTW-MCQ, a collection of manually curated exam multi-choice questions from 21 academic subjects designed to test the broad knowledge and reasoning capabilities of VLMs; and (2) VisTW-Dialogue, an open dialogue benchmark comprising 131 image-question pairs manually created to evaluate VLMs' ability in free-form dialogue generation within Taiwanese cultural contexts. These benchmarks address a critical gap in the evaluation landscape, where existing benchmarks predominantly focus on English or Simplified Chinese, neglecting the unique linguistic and cultural aspects of Traditional Chinese used in regions like Taiwan and Hong Kong. Our analysis reveals significant performance differences across various VLMs and highlights specific challenges in processing Traditional Chinese visual content.