MangaDiT: Reference-Guided Line Art Colorization with Hierarchical Attention in Diffusion Transformers
This work addresses colorization challenges for manga or comic artists, but it is incremental as it builds on existing diffusion models with a novel attention mechanism.
The paper tackles the problem of region-level color inconsistency in reference-guided line art colorization when reference and target images differ in pose or motion, and it achieves superior performance over state-of-the-art methods on benchmark datasets.
Recent advances in diffusion models have significantly improved the performance of reference-guided line art colorization. However, existing methods still struggle with region-level color consistency, especially when the reference and target images differ in character pose or motion. Instead of relying on external matching annotations between the reference and target, we propose to discover semantic correspondences implicitly through internal attention mechanisms. In this paper, we present MangaDiT, a powerful model for reference-guided line art colorization based on Diffusion Transformers (DiT). Our model takes both line art and reference images as conditional inputs and introduces a hierarchical attention mechanism with a dynamic attention weighting strategy. This mechanism augments the vanilla attention with an additional context-aware path that leverages pooled spatial features, effectively expanding the model's receptive field and enhancing region-level color alignment. Experiments on two benchmark datasets demonstrate that our method significantly outperforms state-of-the-art approaches, achieving superior performance in both qualitative and quantitative evaluations.