TRACT: Denoising Diffusion Models with Transitive Closure Time-Distillation
This work addresses the computational inefficiency of diffusion models for practitioners, though it is incremental as it builds on existing time-distillation techniques.
The paper tackles the problem of slow sampling in denoising diffusion models by introducing TRACT, a method that extends binary time-distillation to reduce network calls, achieving up to 2.4x FID improvement and setting new state-of-the-art FID scores of 7.4 on ImageNet64 and 3.8 on CIFAR10 for single-step DDIM.
Denoising Diffusion models have demonstrated their proficiency for generative sampling. However, generating good samples often requires many iterations. Consequently, techniques such as binary time-distillation (BTD) have been proposed to reduce the number of network calls for a fixed architecture. In this paper, we introduce TRAnsitive Closure Time-distillation (TRACT), a new method that extends BTD. For single step diffusion,TRACT improves FID by up to 2.4x on the same architecture, and achieves new single-step Denoising Diffusion Implicit Models (DDIM) state-of-the-art FID (7.4 for ImageNet64, 3.8 for CIFAR10). Finally we tease apart the method through extended ablations. The PyTorch implementation will be released soon.