Score Replacement with Bounded Deviation for Rare Prompt Generation
This addresses a specific issue in image generation for rare concepts, offering an incremental but practical improvement for diffusion model users.
The paper tackles the problem of diffusion models struggling to generate images for rare concepts by proposing a score replacement method with bounded deviation, which adaptively switches from a proxy prompt to the rare prompt based on deviation thresholds, resulting in consistent improvements across multiple models like SDXL and SD3 in automated metrics and human evaluations.
Diffusion models achieve impressive performance in high-fidelity image generation but often struggle with rare concepts that appear infrequently in the training distribution. Prior work attempts to address this issue by prompt switching, where generation begins with a frequent proxy prompt and later transitions to the original rare prompt. However, such designs typically rely on fixed schedules that disregard the model's internal dynamics, making them brittle across prompts and backbones. In this paper, we re-frame rare prompt generation through the lens of score replacement: the denoising trajectory of a rare prompt can be initially guided by the score of a semantically related frequent prompt, which acts as a proxy. However, as the process unfolds, the proxy score gradually diverges from the true rare prompt score. To control this drift, we introduce a bounded deviation criterion that triggers the switch once the deviation exceeds a threshold. This formulation offers both a principled justification and a practical mechanism for rare prompt generation, enabling adaptive switching that can be widely adopted by different models. Extensive experiments across SDXL, SD3, Flux, and Sana confirm that our method consistently improves rare concept synthesis, outperforming strong baselines in both automated metrics and human evaluations.