No Other Representation Component Is Needed: Diffusion Transformers Can Provide Representation Guidance by Themselves
This addresses the need for simpler and more efficient representation learning in diffusion transformers, reducing reliance on complex external frameworks, though it is incremental as it builds on existing diffusion transformer methods.
The paper tackles the problem of needing external representation components for diffusion transformers by proposing Self-Representation Alignment (SRA), a self-distillation method that aligns internal representations during generative training, resulting in consistent performance improvements and competitive results with external priors.
Recent studies have demonstrated that learning a meaningful internal representation can both accelerate generative training and enhance the generation quality of diffusion transformers. However, existing approaches necessitate to either introduce an external and complex representation training framework or rely on a large-scale, pre-trained representation foundation model to provide representation guidance during the original generative training process. In this study, we posit that the unique discriminative process inherent to diffusion transformers enables them to offer such guidance without requiring external representation components. We therefore propose Self-Representation Alignment (SRA), a simple yet straightforward method that obtains representation guidance through a self-distillation manner. Specifically, SRA aligns the output latent representation of the diffusion transformer in the earlier layer with higher noise to that in the later layer with lower noise to progressively enhance the overall representation learning during only the generative training process. Experimental results indicate that applying SRA to DiTs and SiTs yields consistent performance improvements. Moreover, SRA not only significantly outperforms approaches relying on auxiliary, complex representation training frameworks but also achieves performance comparable to methods that are heavily dependent on powerful external representation priors.