CLIPstyler: Image Style Transfer with a Single Text Condition
This addresses a practical problem for users who want to apply styles based on imagination rather than existing images, though it is incremental as it builds on CLIP and existing style transfer methods.
The paper tackles the problem of neural style transfer without requiring a reference style image, enabling style transfer using only a text description. The result is successful image style transfer with realistic textures that reflect semantic query texts, as confirmed by extensive experiments.
Existing neural style transfer methods require reference style images to transfer texture information of style images to content images. However, in many practical situations, users may not have reference style images but still be interested in transferring styles by just imagining them. In order to deal with such applications, we propose a new framework that enables a style transfer `without' a style image, but only with a text description of the desired style. Using the pre-trained text-image embedding model of CLIP, we demonstrate the modulation of the style of content images only with a single text condition. Specifically, we propose a patch-wise text-image matching loss with multiview augmentations for realistic texture transfer. Extensive experimental results confirmed the successful image style transfer with realistic textures that reflect semantic query texts.