Text2Vis: A Challenging and Diverse Benchmark for Generating Multimodal Visualizations from Text
This addresses the problem of evaluating automated data visualization systems for data scientists and AI researchers, though it is incremental as it builds on existing LLM capabilities.
The paper tackles the lack of comprehensive benchmarks for text-to-visualization models by introducing Text2Vis, a diverse benchmark with 1,985 samples covering 20+ chart types, which reveals significant performance gaps among 11 models and proposes a framework that increases GPT-4o's pass rate from 26% to 42%.
Automated data visualization plays a crucial role in simplifying data interpretation, enhancing decision-making, and improving efficiency. While large language models (LLMs) have shown promise in generating visualizations from natural language, the absence of comprehensive benchmarks limits the rigorous evaluation of their capabilities. We introduce Text2Vis, a benchmark designed to assess text-to-visualization models, covering 20+ chart types and diverse data science queries, including trend analysis, correlation, outlier detection, and predictive analytics. It comprises 1,985 samples, each with a data table, natural language query, short answer, visualization code, and annotated charts. The queries involve complex reasoning, conversational turns, and dynamic data retrieval. We benchmark 11 open-source and closed-source models, revealing significant performance gaps, highlighting key challenges, and offering insights for future advancements. To close this gap, we propose the first cross-modal actor-critic agentic framework that jointly refines the textual answer and visualization code, increasing GPT-4o`s pass rate from 26% to 42% over the direct approach and improving chart quality. We also introduce an automated LLM-based evaluation framework that enables scalable assessment across thousands of samples without human annotation, measuring answer correctness, code execution success, visualization readability, and chart accuracy. We release Text2Vis at https://github.com/vis-nlp/Text2Vis.