Sem-CS: Semantic CLIPStyler for Text-Based Image Style Transfer
This work addresses content mismatch and over-stylization issues in text-based image style transfer, improving realism for applications in digital art and media, but it is incremental as it builds on CLIPStyler.
The paper tackled the problem of semantic loss in text-based image style transfer by proposing Semantic CLIPStyler (Sem-CS), which segments content images and uses global foreground and background losses to preserve object semantics, resulting in superior qualitative and quantitative performance as shown by DISTS, NIMA, and user study scores.
CLIPStyler demonstrated image style transfer with realistic textures using only a style text description (instead of requiring a reference style image). However, the ground semantics of objects in the style transfer output is lost due to style spill-over on salient and background objects (content mismatch) or over-stylization. To solve this, we propose Semantic CLIPStyler (Sem-CS), that performs semantic style transfer. Sem-CS first segments the content image into salient and non-salient objects and then transfers artistic style based on a given style text description. The semantic style transfer is achieved using global foreground loss (for salient objects) and global background loss (for non-salient objects). Our empirical results, including DISTS, NIMA and user study scores, show that our proposed framework yields superior qualitative and quantitative performance. Our code is available at github.com/chandagrover/sem-cs.