ChartGaze: Enhancing Chart Understanding in LVLMs with Eye-Tracking Guided Attention Refinement
This addresses the challenge of improving chart understanding in LVLMs for applications like data analysis, though it is incremental as it builds on existing attention mechanisms.
The paper tackled the problem of Large Vision-Language Models (LVLMs) attending to irrelevant regions in chart question answering, which reduces accuracy and interpretability, by proposing a gaze-guided attention refinement method that improved answer accuracy by up to 2.56 percentage points.
Charts are a crucial visual medium for communicating and representing information. While Large Vision-Language Models (LVLMs) have made progress on chart question answering (CQA), the task remains challenging, particularly when models attend to irrelevant regions of the chart. In this work, we present ChartGaze, a new eye-tracking dataset that captures human gaze patterns during chart reasoning tasks. Through a systematic comparison of human and model attention, we find that LVLMs often diverge from human gaze, leading to reduced interpretability and accuracy. To address this, we propose a gaze-guided attention refinement that aligns image-text attention with human fixations. Our approach improves both answer accuracy and attention alignment, yielding gains of up to 2.56 percentage points across multiple models. These results demonstrate the promise of incorporating human gaze to enhance both the reasoning quality and interpretability of chart-focused LVLMs.