Palette Aligned Image Diffusion
This addresses the problem of precise color control in image generation for creative professionals, though it appears to be an incremental improvement over existing conditioning methods.
The paper tackles the problem of conditioning text-to-image diffusion models on user-specified color palettes, which are intuitive but cause ambiguity and instability in generation. The result is a method that consistently outperforms existing approaches in achieving both strong palette adherence and high image quality, as shown through qualitative, quantitative, and user study evaluations.
We introduce the Palette-Adapter, a novel method for conditioning text-to-image diffusion models on a user-specified color palette. While palettes are a compact and intuitive tool widely used in creative workflows, they introduce significant ambiguity and instability when used for conditioning image generation. Our approach addresses this challenge by interpreting palettes as sparse histograms and introducing two scalar control parameters: histogram entropy and palette-to-histogram distance, which allow flexible control over the degree of palette adherence and color variation. We further introduce a negative histogram mechanism that allows users to suppress specific undesired hues, improving adherence to the intended palette under the standard classifier-free guidance mechanism. To ensure broad generalization across the color space, we train on a carefully curated dataset with balanced coverage of rare and common colors. Our method enables stable, semantically coherent generation across a wide range of palettes and prompts. We evaluate our method qualitatively, quantitatively, and through a user study, and show that it consistently outperforms existing approaches in achieving both strong palette adherence and high image quality.