Uni-paint: A Unified Framework for Multimodal Image Inpainting with Pretrained Diffusion Model
This addresses the need for more flexible inpainting tools for users in image editing, though it is incremental as it builds on existing pretrained models.
The paper tackles the problem of limited control in diffusion-based image inpainting by proposing Uni-paint, a unified framework that enables multimodal guidance (e.g., text, strokes, exemplars) without task-specific training, achieving comparable results to single-modal methods.
Recently, text-to-image denoising diffusion probabilistic models (DDPMs) have demonstrated impressive image generation capabilities and have also been successfully applied to image inpainting. However, in practice, users often require more control over the inpainting process beyond textual guidance, especially when they want to composite objects with customized appearance, color, shape, and layout. Unfortunately, existing diffusion-based inpainting methods are limited to single-modal guidance and require task-specific training, hindering their cross-modal scalability. To address these limitations, we propose Uni-paint, a unified framework for multimodal inpainting that offers various modes of guidance, including unconditional, text-driven, stroke-driven, exemplar-driven inpainting, as well as a combination of these modes. Furthermore, our Uni-paint is based on pretrained Stable Diffusion and does not require task-specific training on specific datasets, enabling few-shot generalizability to customized images. We have conducted extensive qualitative and quantitative evaluations that show our approach achieves comparable results to existing single-modal methods while offering multimodal inpainting capabilities not available in other methods. Code will be available at https://github.com/ysy31415/unipaint.