Rethinking Scribble-Guided Image Editing: Generalization, Instruction Adherence, and Multi-Tasking
For researchers and practitioners in interactive image editing, this work provides a systematic analysis and practical strategies to improve multi-task scribble-guided editing, though the improvements are incremental over existing open-source models.
The paper identifies that instruction-level generalization is the main bottleneck in scribble-guided image editing, not image-domain gaps. By proposing a Coverage-then-Realism Curriculum, Multi-Task Mosaicking, and Edit-Focused Loss, they achieve state-of-the-art results on the VIBE benchmark for both single-task and multi-task editing.
Scribble-guided image editing allows users to combine simple scribble annotations with text prompts to specify both where and how an image should be edited, enabling flexible interaction with precise spatial control. However, existing models still exhibit unstable performance under this paradigm, especially in multi-task scenarios. To improve performance, we conduct empirical studies using an open-source editing model and reveal an asymmetry in generalization: instruction-level generalization, including across editing tasks and from single-task to multi-task settings, is more challenging than image-domain generalization, such as from synthetic to real-world images or from mosaicked to regular images. This suggests that the primary bottleneck lies in insufficient learning for diverse editing instructions rather than in the image domain gap. Motivated by this insight, we propose three strategies: (a) a Coverage-then-Realism Curriculum, a two-stage pipeline that first builds large-scale synthetic, instruction-rich data for broad task supervision, then curates a small set of real-world data to refine generation realism; (b) Multi-Task Mosaicking, which constructs multi-task training samples by concatenating single-task examples at nearly zero cost while enabling the learned capability to generalize to non-mosaicked images; and (c) an Edit-Focused Loss, which leverages the changed regions between input and output images in synthetic data to focus training on edited regions, improving both learning efficiency and editing accuracy. With these strategies, we substantially improve both single-task and multi-task scribble-guided editing on the VIBE benchmark, achieving state-of-the-art results. We will publicly release our dataset and model.