Beyond Color and Lines: Zero-Shot Style-Specific Image Variations with Coordinated Semantics
This addresses the challenge of creating stylized images without losing semantic accuracy, which is useful for artists and designers, though it appears incremental as it builds on existing vision-language and diffusion models.
The paper tackles the problem of generating style-specific image variations while preserving semantic content by proposing a zero-shot scheme that transforms image-to-image generation into an image-to-text-to-image pipeline using vision-language and diffusion models, achieving highly plausible results with high-fidelity semantics.
Traditionally, style has been primarily considered in terms of artistic elements such as colors, brushstrokes, and lighting. However, identical semantic subjects, like people, boats, and houses, can vary significantly across different artistic traditions, indicating that style also encompasses the underlying semantics. Therefore, in this study, we propose a zero-shot scheme for image variation with coordinated semantics. Specifically, our scheme transforms the image-to-image problem into an image-to-text-to-image problem. The image-to-text operation employs vision-language models e.g., BLIP) to generate text describing the content of the input image, including the objects and their positions. Subsequently, the input style keyword is elaborated into a detailed description of this style and then merged with the content text using the reasoning capabilities of ChatGPT. Finally, the text-to-image operation utilizes a Diffusion model to generate images based on the text prompt. To enable the Diffusion model to accommodate more styles, we propose a fine-tuning strategy that injects text and style constraints into cross-attention. This ensures that the output image exhibits similar semantics in the desired style. To validate the performance of the proposed scheme, we constructed a benchmark comprising images of various styles and scenes and introduced two novel metrics. Despite its simplicity, our scheme yields highly plausible results in a zero-shot manner, particularly for generating stylized images with high-fidelity semantics.