Multi-Step Consistency Models: Fast Generation with Theoretical Guarantees
This work addresses a theoretical gap for researchers and practitioners in generative AI, offering rigorous guarantees for fast generation models, though it is incremental in building on existing consistency model frameworks.
The paper tackles the lack of theoretical justification for the speed of consistency models in generative AI by providing a theoretical analysis that shows achieving a KL divergence of order O(ε²) requires only O(log(d/ε)) iterations with constant step size, and under non-smooth data distributions, it scales as O(d log(d/ε)), with best-in-class convergence rates compared to existing methods.
Consistency models have recently emerged as a compelling alternative to traditional SDE-based diffusion models. They offer a significant acceleration in generation by producing high-quality samples in very few steps. Despite their empirical success, a proper theoretic justification for their speed-up is still lacking. In this work, we address the gap by providing a theoretical analysis of consistency models capable of mapping inputs at a given time to arbitrary points along the reverse trajectory. We show that one can achieve a KL divergence of order $ O(\varepsilon^2) $ using only $ O\left(\log\left(\frac{d}{\varepsilon}\right)\right) $ iterations with a constant step size. Additionally, under minimal assumptions on the data distribution (non smooth case) an increasingly common setting in recent diffusion model analyses we show that a similar KL convergence guarantee can be obtained, with the number of steps scaling as $ O\left(d \log\left(\frac{d}{\varepsilon}\right)\right) $. Going further, we also provide a theoretical analysis for estimation of such consistency models, concluding that accurate learning is feasible using small discretization steps, both in smooth and non-smooth settings. Notably, our results for the non-smooth case yield best in class convergence rates compared to existing SDE or ODE based analyses under minimal assumptions.