MIND-Edit: MLLM Insight-Driven Editing via Language-Vision Projection
This addresses the challenge of insufficient alignment between textual semantics and visual outcomes in image editing for users needing fine-grained edits, though it appears incremental as it builds on existing MLLM-based methods.
The paper tackles the problem of achieving high precision and semantic accuracy in image editing by proposing MIND-Edit, an end-to-end framework that integrates a pretrained diffusion model with a multimodal large language model (MLLM) to optimize text instructions and leverage visual insights, resulting in outperforming state-of-the-art methods in quantitative metrics and visual quality.
Recent advances in AI-generated content (AIGC) have significantly accelerated image editing techniques, driving increasing demand for diverse and fine-grained edits. Despite these advances, existing image editing methods still face challenges in achieving high precision and semantic accuracy in complex scenarios. Recent studies address this issue by incorporating multimodal large language models (MLLMs) into image editing pipelines. However, current MLLM-based methods mainly rely on interpreting textual instructions, leaving the intrinsic visual understanding of large models largely unexplored, thus resulting in insufficient alignment between textual semantics and visual outcomes. To overcome these limitations, we propose MIND-Edit, an end-to-end image-editing framework integrating pretrained diffusion model with MLLM. MIND-Edit introduces two complementary strategies: (1) a text instruction optimization strategy that clarifies ambiguous user instructions based on semantic reasoning from the MLLM, and (2) an MLLM insight-driven editing strategy that explicitly leverages the intrinsic visual understanding capability of the MLLM to infer editing intent and guide the diffusion process via generated visual embeddings. Furthermore, we propose a joint training approach to effectively integrate both strategies, allowing them to reinforce each other for more accurate instruction interpretation and visually coherent edits aligned with user intent. Extensive experiments demonstrate that MIND-Edit outperforms state-of-the-art image editing methods in both quantitative metrics and visual quality, particularly under complex and challenging scenarios.