Multimodal Semantic-Aware Automatic Colorization with Diffusion Prior
This work addresses the challenge of generating realistic and semantically accurate colorized images for applications in photography and visual media, representing an incremental improvement over existing methods.
The authors tackled the problem of automatic grayscale image colorization, which often suffers from incorrect semantic colors and unsaturated results, by proposing a pipeline that leverages a diffusion prior and multimodal semantic priors to generate saturated colors with plausible semantics, achieving superior perceptual realism and human preference.
Colorizing grayscale images offers an engaging visual experience. Existing automatic colorization methods often fail to generate satisfactory results due to incorrect semantic colors and unsaturated colors. In this work, we propose an automatic colorization pipeline to overcome these challenges. We leverage the extraordinary generative ability of the diffusion prior to synthesize color with plausible semantics. To overcome the artifacts introduced by the diffusion prior, we apply the luminance conditional guidance. Moreover, we adopt multimodal high-level semantic priors to help the model understand the image content and deliver saturated colors. Besides, a luminance-aware decoder is designed to restore details and enhance overall visual quality. The proposed pipeline synthesizes saturated colors while maintaining plausible semantics. Experiments indicate that our proposed method considers both diversity and fidelity, surpassing previous methods in terms of perceptual realism and gain most human preference.