PrefPaint: Enhancing Image Inpainting through Expert Human Feedback
This addresses reliability issues in medical diagnosis by improving inpainting accuracy for domains like polyp imaging, though it is incremental as it builds on existing methods with human feedback.
The paper tackles the problem of inaccurate image inpainting in specialized fields like medical imaging by proposing PrefPaint, which incorporates expert human feedback into training Stable Diffusion Inpainting, resulting in reduced visual inconsistencies and more realistic polyp images.
Inpainting, the process of filling missing or corrupted image parts, has broad applications, including medical imaging. However, in specialized fields like medical polyps imaging, where accuracy and reliability are critical, inpainting models can generate inaccurate images, leading to significant errors in medical diagnosis and treatment. To ensure reliability, medical images should be annotated by experts like oncologists for effective model training. We propose PrefPaint, an approach that incorporates human feedback into the training process of Stable Diffusion Inpainting, bypassing the need for computationally expensive reward models. In addition, we develop a web-based interface streamlines training, fine-tuning, and inference. This interactive interface provides a smooth and intuitive user experience, making it easier to offer feedback and manage the fine-tuning process. User study on various domains shows that PrefPaint outperforms existing methods, reducing visual inconsistencies and improving image rendering, particularly in medical contexts, where our model generates more realistic polyps images.