Program Synthesis Benchmark for Visual Programming in XLogoOnline Environment
This provides a new benchmark for assessing multimodal models in visual programming, addressing a gap in evaluating combined skills, though it is incremental as it builds on existing environments and methods.
The authors tackled the problem of evaluating multimodal models on tasks requiring combined skills like spatial planning and programming by curating a benchmark based on XLogoOnline, finding that GPT-4V and Llama3-70B achieved only 20% and 2.35% success rates, but fine-tuning with synthetic data and curriculum learning improved performance.
Large language and multimodal models have shown remarkable success on various benchmarks focused on specific skills such as general-purpose programming, math word problem-solving, and visual question answering. However, it is unclear how well these models perform on tasks that require a combination of these skills. In this paper, we curate a novel program synthesis benchmark based on the real-world tasks in the XLogoOnline visual programming environment. Each task requires a combination of different skills such as spatial planning, basic programming, and logical reasoning. Our evaluation shows that current state-of-the-art models like GPT-4V and Llama3-70B struggle to solve these tasks, achieving only 20% and 2.35% success rates, respectively. Next, we develop a fine-tuning pipeline to boost the performance of models by leveraging a large-scale synthetic training dataset with over 80,000 tasks. Moreover, we showcase how emulator-driven feedback can be used to design a curriculum over training data distribution, through which a fine-tuned Llama3-8B drastically outperforms GPT-4V and Llama3-70B models. Finally, we provide an in-depth failure analysis to understand the limitations of different models. We will publicly release the benchmark for future research on program synthesis in visual programming.