GuessBench: Sensemaking Multimodal Creativity in the Wild
This addresses the challenge of assessing VLMs in noisy, real-world creative tasks for AI researchers, though it is incremental as it builds on existing benchmark methodologies.
The paper tackles the problem of evaluating Vision Language Models (VLMs) on modeling human creativity by introducing GuessBench, a benchmark sourced from an online Minecraft minigame, and finds that even state-of-the-art models like GPT-4o are incorrect on 34% of instances, with fine-tuning on reasoning traces improving visual perception tasks by 15.36% on average.
We propose GuessBench, a novel benchmark that evaluates Vision Language Models (VLMs) on modeling the pervasive, noisy, and pluralistic human creativity. GuessBench sources data from "Guess the Build", an online multiplayer Minecraft minigame where one player constructs a Minecraft build given a concept (e.g. caterpillar) and others try to guess it with natural language hints, presenting a pristine testbed for sensemaking creativity in the wild with VLMs acting as guessers. We curate 1500 images from the actual gameplay and design 2000 problems spanning static and dynamic image settings, natural language hints of varying completeness, and more. Extensive experiments with six open/API VLMs and five reasoning enhancement approaches demonstrate that GuessBench presents a uniquely challenging task in creativity modeling: even the start-of-the-art GPT-4o is incorrect on 34% of instances, while we observe a huge performance gap (13.87% vs. 53.93% on average) between open and API models. When used as a resource to improve VLMs, fine-tuning on the reasoning traces for GuessBench problems improves visual perception tasks by 15.36% on average. Further analysis reveals that VLM performance in creativity sensemaking correlates with the frequency of the concept in training data, while the accuracy drops sharply for concepts in underrepresented cultural contexts and low-resource languages.