Hyper-SD: Trajectory Segmented Consistency Model for Efficient Image Synthesis
This work addresses the problem of slow image synthesis for users of diffusion models by providing an efficient method with incremental improvements over existing distillation techniques.
The paper tackles the computational inefficiency of multi-step inference in Diffusion Models by proposing Hyper-SD, a framework that combines trajectory preservation and reformulation to achieve near-lossless performance with fewer steps, resulting in state-of-the-art performance for 1 to 8 steps, such as surpassing SDXL-Lightning by +0.68 in CLIP Score and +0.51 in Aes Score in 1-step inference.
Recently, a series of diffusion-aware distillation algorithms have emerged to alleviate the computational overhead associated with the multi-step inference process of Diffusion Models (DMs). Current distillation techniques often dichotomize into two distinct aspects: i) ODE Trajectory Preservation; and ii) ODE Trajectory Reformulation. However, these approaches suffer from severe performance degradation or domain shifts. To address these limitations, we propose Hyper-SD, a novel framework that synergistically amalgamates the advantages of ODE Trajectory Preservation and Reformulation, while maintaining near-lossless performance during step compression. Firstly, we introduce Trajectory Segmented Consistency Distillation to progressively perform consistent distillation within pre-defined time-step segments, which facilitates the preservation of the original ODE trajectory from a higher-order perspective. Secondly, we incorporate human feedback learning to boost the performance of the model in a low-step regime and mitigate the performance loss incurred by the distillation process. Thirdly, we integrate score distillation to further improve the low-step generation capability of the model and offer the first attempt to leverage a unified LoRA to support the inference process at all steps. Extensive experiments and user studies demonstrate that Hyper-SD achieves SOTA performance from 1 to 8 inference steps for both SDXL and SD1.5. For example, Hyper-SDXL surpasses SDXL-Lightning by +0.68 in CLIP Score and +0.51 in Aes Score in the 1-step inference.