DiverseDiT++: Quantifying, Analyzing, and Promoting Representation Diversity in Diffusion Transformers
This work provides a new metric and optimization strategy for researchers and practitioners working on Diffusion Transformers, offering a principled guide for improving model performance and understanding internal mechanisms.
This paper introduces the Weighted Diversity Score (WDS) to quantify representation diversity in Diffusion Transformers (DiTs), finding a strong correlation between WDS and synthesis quality (Pearson's r=-0.869 with log(FID)). Based on this, they propose DiverseDiT++, a framework that uses long residual connections and a representation diversity loss to promote diverse representation learning, leading to consistent performance gains and faster convergence on ImageNet.
Recent advances in Diffusion Transformers (DiTs) have enabled remarkable progress in visual synthesis, benefiting from their superior scalability. To facilitate DiTs' capability of capturing meaningful internal representations, recent works such as REPA incorporate external pretrained encoders for representation alignment. However, the underlying mechanisms governing representation learning within DiTs remain poorly understood in the community. To this end, this paper first presents a systematic analysis of the representation dynamics of DiTs via quantifying the diversity of block-wise representations. Specifically, we introduce a novel metric, termed the Weighted Diversity Score (WDS), to measure the representational discrepancies across different blocks. Through extensive investigations on the evolution and influence of internal representations under various settings, we reveal that representation diversity across blocks is a critical factor for effective representation learning in DiTs. More importantly, WDS exhibits a strong correlation with synthesis quality across diverse settings, model scales, and training stages (Pearson's $r=-0.869$ with $\log(\text{FID})$), suggesting its potential as an indicator to reflect model performance and a principled guide for model optimization. Based on this key finding, we propose DiverseDiT++, a novel framework that explicitly promotes diverse representation learning. Concretely, our method incorporates long residual connections to diversify input representations across blocks and a representation diversity loss to encourage blocks to learn distinct features. Extensive experiments on ImageNet $256\times256$ and $512\times512$ demonstrate that our DiverseDiT++ yields consistent performance gains and convergence acceleration when applied to different backbones with various sizes,...