Assessing Y-Axis Influence: Bias in Multimodal Language Models on Chart-to-Table Translation
For researchers and practitioners using multimodal language models for chart understanding, this work systematically exposes previously unexamined y-axis biases that can affect model reliability.
The paper identifies and analyzes biases in chart-to-table translation caused by imbalances in y-axis features across public datasets, showing significant performance variations in five multimodal language models. Key findings include biases related to digit length, number of ticks, value range, and tick format, with prompting improvements for some models.
Chart-to-table translation converts chart images into structured tabular data. Accurate translation is crucial for Multimodal Language Model (MLM) to answer complex queries. We observe imbalances in the number of images across different aspects of the y-axis information in public chart datasets. Such imbalances can introduce unintended biases, causing uneven MLM performance. Previous works have not systematically examined these biases. To address this gap, we propose a new framework, FairChart2Table, for analyzing y-axis-related bias on five state-of-the-art models. Key Findings: (1) There are significant y-axis biases related to the digit length of the major tick values, the number of major ticks, the range of values, and the tick value format (e.g., abbreviation or scientific format). (2) The number of legends/entities in chart images impacts MLM performance. (3) Prompting MLM with y-axis information can significantly enhance the performance for some MLMs.