Fill in the ____ (a Diffusion-based Image Inpainting Pipeline)
This work tackles the problem of controllable image generation for users needing precise inpainting, though it appears incremental as it builds on existing diffusion-based methods.
The paper addresses a critical gap in image inpainting models by enabling prompt-based control over generated content, proposing multiple approaches to implement this functionality and evaluating them qualitatively for high-quality, instruction-following results.
Image inpainting is the process of taking an image and generating lost or intentionally occluded portions. Inpainting has countless applications including restoring previously damaged pictures, restoring the quality of images that have been degraded due to compression, and removing unwanted objects/text. Modern inpainting techniques have shown remarkable ability in generating sensible completions for images with mask occlusions. In our paper, an overview of the progress of inpainting techniques will be provided, along with identifying current leading approaches, focusing on their strengths and weaknesses. A critical gap in these existing models will be addressed, focusing on the ability to prompt and control what exactly is generated. We will additionally justify why we think this is the natural next progressive step that inpainting models must take, and provide multiple approaches to implementing this functionality. Finally, we will evaluate the results of our approaches by qualitatively checking whether they generate high-quality images that correctly inpaint regions with the objects that they are instructed to produce.