StructChart: On the Schema, Metric, and Augmentation for Visual Chart Understanding
This work addresses the challenge of integrating chart perception and reasoning for researchers and practitioners in data visualization and AI, representing an incremental advance by unifying existing tasks with a novel representation.
The authors tackled the problem of chart understanding by proposing StructChart, a unified framework for chart perception and reasoning that uses Structured Triplet Representations, achieving improved performance across various tasks with enhanced efficiency and diversity through LLM-based augmentation.
Charts are common in literature across various scientific fields, conveying rich information easily accessible to readers. Current chart-related tasks focus on either chart perception that extracts information from the visual charts, or chart reasoning given the extracted data, e.g. in a tabular form. In this paper, we introduce StructChart, a novel framework that leverages Structured Triplet Representations (STR) to achieve a unified and label-efficient approach to chart perception and reasoning tasks, which is generally applicable to different downstream tasks, beyond the question-answering task as specifically studied in peer works. Specifically, StructChart first reformulates the chart data from the tubular form (linearized CSV) to STR, which can friendlily reduce the task gap between chart perception and reasoning. We then propose a Structuring Chart-oriented Representation Metric (SCRM) to quantitatively evaluate the chart perception task performance. To augment the training, we further explore the potential of Large Language Models (LLMs) to enhance the diversity in both chart visual style and statistical information. Extensive experiments on various chart-related tasks demonstrate the effectiveness and potential of a unified chart perception-reasoning paradigm to push the frontier of chart understanding.