The Image Reconstruction Game: Drawing Common Ground Through Iterative Multimodal Dialogue
For researchers in multimodal AI, this benchmark provides a way to measure common ground accumulation, but the results are incremental and the automated evaluation requires human recalibration.
The paper introduces the Image Reconstruction Game, a benchmark for evaluating multimodal dialogue between vision-language models and image generators. It finds that the describer model dominates reconstruction quality, while the generator determines the benefit of iterative refinement, with mathematical and geometric images being the hardest.
We introduce the Image Reconstruction Game, a fully automated benchmark in which a vision-language model issues corrective instructions to an image generator across multiple turns, making accumulated common ground directly observable as a rendered image. Benchmarking two Describer models crossed with two Generator models across seven image categories, we find that the describer is the dominant factor in reconstruction quality, while the generator determines whether iterative refinement helps or hurts. Mathematical and geometric images pose the greatest challenge. The describer's token budget strongly affects convergence: shorter budgets yield sparser first renderings with more room for visible improvement, while longer budgets raise absolute quality but leave less to fix. Stronger describers use a richer correction vocabulary spanning spatial, numeric, and structural categories, while weaker describers concentrate on surface properties and tend to stop after a few turns. Human validation shows that the best automated judge reaches only slight-to-fair agreement with human preferences, and automated scores require human recalibration to be used reliably.