Soulstyler: Using Large Language Model to Guide Image Style Transfer for Target Object
This addresses a limitation in image style transfer for users needing precise object-level stylization without stylized reference images, though it is incremental as it builds on existing CLIP and language model methods.
The paper tackles the problem of stylizing specific objects in an image without affecting background regions, using textual descriptions to guide the process, and achieves accurate style transfer on target objects as demonstrated experimentally.
Image style transfer occupies an important place in both computer graphics and computer vision. However, most current methods require reference to stylized images and cannot individually stylize specific objects. To overcome this limitation, we propose the "Soulstyler" framework, which allows users to guide the stylization of specific objects in an image through simple textual descriptions. We introduce a large language model to parse the text and identify stylization goals and specific styles. Combined with a CLIP-based semantic visual embedding encoder, the model understands and matches text and image content. We also introduce a novel localized text-image block matching loss that ensures that style transfer is performed only on specified target objects, while non-target regions remain in their original style. Experimental results demonstrate that our model is able to accurately perform style transfer on target objects according to textual descriptions without affecting the style of background regions. Our code will be available at https://github.com/yisuanwang/Soulstyler.