Mask-ControlNet: Higher-Quality Image Generation with An Additional Mask Prompt
This work addresses a specific challenge in image generation for applications requiring detailed control, but it is incremental as it builds on existing diffusion models.
The paper tackles the problem of limited accuracy in text-to-image generation when foreground-background relationships are complex by introducing Mask-ControlNet, which uses an additional mask prompt from reference images. Experiments show it achieves higher fidelity and better image quality compared to previous methods.
Text-to-image generation has witnessed great progress, especially with the recent advancements in diffusion models. Since texts cannot provide detailed conditions like object appearance, reference images are usually leveraged for the control of objects in the generated images. However, existing methods still suffer limited accuracy when the relationship between the foreground and background is complicated. To address this issue, we develop a framework termed Mask-ControlNet by introducing an additional mask prompt. Specifically, we first employ large vision models to obtain masks to segment the objects of interest in the reference image. Then, the object images are employed as additional prompts to facilitate the diffusion model to better understand the relationship between foreground and background regions during image generation. Experiments show that the mask prompts enhance the controllability of the diffusion model to maintain higher fidelity to the reference image while achieving better image quality. Comparison with previous text-to-image generation methods demonstrates our method's superior quantitative and qualitative performance on the benchmark datasets.