Latent Forcing: Reordering the Diffusion Trajectory for Pixel-Space Image Generation
This addresses the efficiency and modeling limitations of latent diffusion for image generation, offering an incremental improvement by modifying existing architectures.
The paper tackles the problem of latent diffusion models losing end-to-end modeling benefits by discarding information during encoding and requiring separate decoders, proposing Latent Forcing to reorder the denoising trajectory for joint latent-pixel processing, achieving a new state-of-the-art for diffusion transformer-based pixel generation on ImageNet at their compute scale.
Latent diffusion models excel at generating high-quality images but lose the benefits of end-to-end modeling. They discard information during image encoding, require a separately trained decoder, and model an auxiliary distribution to the raw data. In this paper, we propose Latent Forcing, a simple modification to existing architectures that achieves the efficiency of latent diffusion while operating on raw natural images. Our approach orders the denoising trajectory by jointly processing latents and pixels with separately tuned noise schedules. This allows the latents to act as a scratchpad for intermediate computation before high-frequency pixel features are generated. We find that the order of conditioning signals is critical, and we analyze this to explain differences between REPA distillation in the tokenizer and the diffusion model, conditional versus unconditional generation, and how tokenizer reconstruction quality relates to diffusability. Applied to ImageNet, Latent Forcing achieves a new state-of-the-art for diffusion transformer-based pixel generation at our compute scale.