VizExtract: Automatic Relation Extraction from Data Visualizations
This addresses the challenge of information retrieval from visualizations for researchers and fact-checkers, but it is incremental as it builds on existing computer vision methods for chart analysis.
The paper tackles the problem of automatically extracting compared variables from statistical charts, such as line graphs and scatter plots, to aid in search and data extraction. It achieves 87.5% accuracy in controlled experiments and up to 84.7% accuracy on real-world datasets like FigureQA.
Visual graphics, such as plots, charts, and figures, are widely used to communicate statistical conclusions. Extracting information directly from such visualizations is a key sub-problem for effective search through scientific corpora, fact-checking, and data extraction. This paper presents a framework for automatically extracting compared variables from statistical charts. Due to the diversity and variation of charting styles, libraries, and tools, we leverage a computer vision based framework to automatically identify and localize visualization facets in line graphs, scatter plots, or bar graphs and can include multiple series per graph. The framework is trained on a large synthetically generated corpus of matplotlib charts and we evaluate the trained model on other chart datasets. In controlled experiments, our framework is able to classify, with 87.5% accuracy, the correlation between variables for graphs with 1-3 series per graph, varying colors, and solid line styles. When deployed on real-world graphs scraped from the internet, it achieves 72.8% accuracy (81.2% accuracy when excluding "hard" graphs). When deployed on the FigureQA dataset, it achieves 84.7% accuracy.