Bridging Text and Image for Artist Style Transfer via Contrastive Learning
This enables more flexible and convenient artistic style transfer for users by allowing text descriptions instead of reference images, though it is incremental in leveraging existing image-text models.
The paper tackles the problem of requiring style images for image style transfer by proposing a text-driven method using contrastive learning, achieving state-of-the-art results with rendering times of 0.03s for 512x512 images.
Image style transfer has attracted widespread attention in the past few years. Despite its remarkable results, it requires additional style images available as references, making it less flexible and inconvenient. Using text is the most natural way to describe the style. More importantly, text can describe implicit abstract styles, like styles of specific artists or art movements. In this paper, we propose a Contrastive Learning for Artistic Style Transfer (CLAST) that leverages advanced image-text encoders to control arbitrary style transfer. We introduce a supervised contrastive training strategy to effectively extract style descriptions from the image-text model (i.e., CLIP), which aligns stylization with the text description. To this end, we also propose a novel and efficient adaLN based state space models that explore style-content fusion. Finally, we achieve a text-driven image style transfer. Extensive experiments demonstrate that our approach outperforms the state-of-the-art methods in artistic style transfer. More importantly, it does not require online fine-tuning and can render a 512x512 image in 0.03s.