WEAVE: Unleashing and Benchmarking the In-context Interleaved Comprehension and Generation
This addresses the problem of evaluating and improving multi-turn, context-aware image generation and editing for the multimodal AI community, though it is incremental as it builds on existing unified multimodal models.
The paper tackles the lack of datasets and benchmarks for multi-turn, context-dependent image creation and editing in unified multimodal models by introducing WEAVE, a suite consisting of WEAVE-100k (100K interleaved samples with 370K dialogue turns and 500K images) and WEAVEBench (100 tasks on 480 images), which enables training for capabilities like visual memory and exposes limitations in current approaches.
Recent advances in unified multimodal models (UMMs) have enabled impressive progress in visual comprehension and generation. However, existing datasets and benchmarks focus primarily on single-turn interactions, failing to capture the multi-turn, context-dependent nature of real-world image creation and editing. To address this gap, we present WEAVE, the first suite for in-context interleaved cross-modality comprehension and generation. Our suite consists of two complementary parts. WEAVE-100k is a large-scale dataset of 100K interleaved samples spanning over 370K dialogue turns and 500K images, covering comprehension, editing, and generation tasks that require reasoning over historical context. WEAVEBench is a human-annotated benchmark with 100 tasks based on 480 images, featuring a hybrid VLM judger evaluation framework based on both the reference image and the combination of the original image with editing instructions that assesses models' abilities in multi-turn generation, visual memory, and world-knowledge reasoning across diverse domains. Experiments demonstrate that training on WEAVE-100k enables vision comprehension, image editing, and comprehension-generation collaboration capabilities. Furthermore, it facilitates UMMs to develop emergent visual-memory capabilities, while extensive evaluations on WEAVEBench expose the persistent limitations and challenges of current approaches in multi-turn, context-aware image generation and editing. We believe WEAVE provides a view and foundation for studying in-context interleaved comprehension and generation for multi-modal community.