L-CAD: Language-based Colorization with Any-level Descriptions using Diffusion Priors
This work addresses a practical limitation in user-guided colorization tools for digital artists and designers, offering an incremental improvement by handling ambiguous inputs more effectively.
The paper tackles the problem of language-based image colorization when users provide incomplete or ambiguous color descriptions, proposing a unified model that leverages diffusion priors to handle any-level descriptions and achieves instance-aware colorization with superior performance over existing methods.
Language-based colorization produces plausible and visually pleasing colors under the guidance of user-friendly natural language descriptions. Previous methods implicitly assume that users provide comprehensive color descriptions for most of the objects in the image, which leads to suboptimal performance. In this paper, we propose a unified model to perform language-based colorization with any-level descriptions. We leverage the pretrained cross-modality generative model for its robust language understanding and rich color priors to handle the inherent ambiguity of any-level descriptions. We further design modules to align with input conditions to preserve local spatial structures and prevent the ghosting effect. With the proposed novel sampling strategy, our model achieves instance-aware colorization in diverse and complex scenarios. Extensive experimental results demonstrate our advantages of effectively handling any-level descriptions and outperforming both language-based and automatic colorization methods. The code and pretrained models are available at: https://github.com/changzheng123/L-CAD.