Training-Free Style and Content Transfer by Leveraging U-Net Skip Connections in Stable Diffusion
This work addresses the need for training-free image editing and style transfer, offering a novel approach for users in computer vision and graphics, though it is incremental as it builds on existing diffusion model components.
The paper tackled the problem of style and content transfer in images by analyzing the role of U-Net skip connections in Stable Diffusion, finding that the third encoder block carries spatial information for content while the decoding layer handles style, and demonstrated that their SkipInject method achieves the best content alignment and structural preservation compared to state-of-the-art methods.
Recent advances in diffusion models for image generation have led to detailed examinations of several components within the U-Net architecture for image editing. While previous studies have focused on the bottleneck layer (h-space), cross-attention, self-attention, and decoding layers, the overall role of the skip connections of the U-Net itself has not been specifically addressed. We conduct thorough analyses on the role of the skip connections and find that the residual connections passed by the third encoder block carry most of the spatial information of the reconstructed image, splitting the content from the style, passed by the remaining stream in the opposed decoding layer. We show that injecting the representations from this block can be used for text-based editing, precise modifications, and style transfer. We compare our method, SkipInject, to state-of-the-art style transfer and image editing methods and demonstrate that our method obtains the best content alignment and optimal structural preservation tradeoff.