Speedrunning ImageNet Diffusion
This work addresses the computational cost of training diffusion models for image generation, providing a more efficient baseline for researchers, though it is incremental in combining existing techniques.
The paper tackles the problem of inefficient training of diffusion transformers by introducing SR-DiT, a framework that combines token routing, architectural improvements, and training modifications, achieving FID 3.49 and KDD 0.319 on ImageNet-256 with a 140M parameter model at 400K iterations, comparable to larger models.
Recent advances have significantly improved the training efficiency of diffusion transformers. However, these techniques have largely been studied in isolation, leaving unexplored the potential synergies from combining multiple approaches. We present SR-DiT (Speedrun Diffusion Transformer), a framework that systematically integrates token routing, architectural improvements, and training modifications on top of representation alignment. Our approach achieves FID 3.49 and KDD 0.319 on ImageNet-256 using only a 140M parameter model at 400K iterations without classifier-free guidance - comparable to results from 685M parameter models trained significantly longer. To our knowledge, this is a state-of the-art result at this model size. Through extensive ablation studies, we identify which technique combinations are most effective and document both synergies and incompatibilities. We release our framework as a computationally accessible baseline for future research.