DiffUTE: Universal Text Editing Diffusion Model
This addresses a specific bottleneck in text rendering for image editing, enabling more accurate and controllable modifications in real-world applications, though it appears incremental as it builds on existing diffusion models.
The paper tackles the problem of diffusion models struggling to render correct text and text style in language-guided image editing by proposing DiffUTE, a universal self-supervised text editing diffusion model that replaces or modifies words in source images while maintaining realistic appearance, achieving impressive performance and high fidelity on in-the-wild images.
Diffusion model based language-guided image editing has achieved great success recently. However, existing state-of-the-art diffusion models struggle with rendering correct text and text style during generation. To tackle this problem, we propose a universal self-supervised text editing diffusion model (DiffUTE), which aims to replace or modify words in the source image with another one while maintaining its realistic appearance. Specifically, we build our model on a diffusion model and carefully modify the network structure to enable the model for drawing multilingual characters with the help of glyph and position information. Moreover, we design a self-supervised learning framework to leverage large amounts of web data to improve the representation ability of the model. Experimental results show that our method achieves an impressive performance and enables controllable editing on in-the-wild images with high fidelity. Our code will be avaliable in \url{https://github.com/chenhaoxing/DiffUTE}.