MixSA: Training-free Reference-based Sketch Extraction via Mixture-of-Self-Attention
This addresses the need for versatile and practical sketch extraction tools for users in digital art or design, though it is incremental as it builds on existing diffusion and self-attention techniques.
The paper tackled the problem of sketch extraction methods requiring extensive training or lacking style versatility by introducing MixSA, a training-free method that uses diffusion priors and mixture-of-self-attention to integrate reference sketch styles, resulting in superior performance in sketch quality and flexibility as evaluated by perceptual metrics.
Current sketch extraction methods either require extensive training or fail to capture a wide range of artistic styles, limiting their practical applicability and versatility. We introduce Mixture-of-Self-Attention (MixSA), a training-free sketch extraction method that leverages strong diffusion priors for enhanced sketch perception. At its core, MixSA employs a mixture-of-self-attention technique, which manipulates self-attention layers by substituting the keys and values with those from reference sketches. This allows for the seamless integration of brushstroke elements into initial outline images, offering precise control over texture density and enabling interpolation between styles to create novel, unseen styles. By aligning brushstroke styles with the texture and contours of colored images, particularly in late decoder layers handling local textures, MixSA addresses the common issue of color averaging by adjusting initial outlines. Evaluated with various perceptual metrics, MixSA demonstrates superior performance in sketch quality, flexibility, and applicability. This approach not only overcomes the limitations of existing methods but also empowers users to generate diverse, high-fidelity sketches that more accurately reflect a wide range of artistic expressions.