What You See Is What You Get: Observation-Aligned Supervision for Chart-to-Code Generation
For researchers and practitioners in chart-to-code generation, this work highlights the importance of supervision targets that respect identifiability, offering a simple rewriting framework that improves model faithfulness without requiring more data or complex algorithms.
The paper identifies that chart-to-code generation models are trained to reproduce latent raw data that cannot be inferred from the chart image, leading to hallucination. By replacing such targets with observation-aligned quantities (e.g., bin weights for histograms), they achieve consistent improvements in observable value recovery across multiple VLMs on ChartMimic and ChartX.
Chart-to-code generation is commonly trained with supervised fine-tuning on reference plotting scripts, implicitly treating the gold code as a fully observable target. We argue that this assumption is often invalid: many chart programs contain latent raw variables that cannot be uniquely recovered from the rendered image. For example, a boxplot exposes summary statistics rather than original samples, a pie chart reveals proportions rather than arbitrary raw values, and a histogram shows bin-level mass rather than individual observations. Supervising models to reproduce such non-identifiable quantities encourages hallucination and over-specified code generation. We introduce Observation-Aligned supervision, a rewriting framework that replaces latent raw-data targets with quantities constrained by the visual observation: box statistics for boxplots, wedge percentages for pie charts, and bin weights for histograms. Applying this framework to chart-to-code training data from two sources, we obtain the Observation-Aligned supervision target data. Experiments across multiple VLMs on ChartMimic and ChartX demonstrate consistent improvements in observable value recovery, including under both-executable evaluation. Our results suggest that improving chart-to-code models requires not only more data or advanced learning objectives or algorithms, but also supervision targets that respect what is identifiable from the chart image.