TMPDiff: Temporal Mixed-Precision for Diffusion Models
This addresses the problem of slow inference in text-to-image generation for users needing faster deployment, but it is incremental as it builds on existing quantization methods.
The paper tackles the high inference latency of diffusion models by proposing TMPDiff, a temporal mixed-precision framework that assigns different numeric precisions to denoising timesteps, achieving 10 to 20% improvement in perceptual quality and 90% SSIM relative to full-precision at a 2.5x speedup.
Diffusion models are the go-to method for Text-to-Image generation, but their iterative denoising processes has high inference latency. Quantization reduces compute time by using lower bitwidths, but applies a fixed precision across all denoising timesteps, leaving an entire optimization axis unexplored. We propose TMPDiff, a temporal mixed-precision framework for diffusion models that assigns different numeric precision to different denoising timesteps. We hypothesize that quantization errors accumulate additively across timesteps, which we then validate experimentally. Based on our observations, we develop an adaptive bisectioning-based algorithm, which assigns per-step precisions with linear evaluation complexity, reducing an otherwise exponential search problem. Across four state-of-the-art diffusion models and three datasets, TMPDiff consistently outperforms uniform-precision baselines at matched speedup, achieving 10 to 20% improvement in perceptual quality. On FLUX.1-dev, TMPDiff achieves 90% SSIM relative to the full-precision model at a speedup of 2.5x over 16-bit inference.