Dynamic Differential Linear Attention: Enhancing Linear Diffusion Transformer for High-Quality Image Generation
This work addresses a scalability bottleneck for high-quality image generation in diffusion transformers, offering a novel linear attention method that improves performance without quadratic cost, though it is incremental as it builds on existing LiT frameworks.
The paper tackles the problem of generative performance degradation in linear diffusion transformers (LiTs) due to over-smoothed attention weights, and introduces Dynamic Differential Linear Attention (DyDiLA) to enhance LiTs, resulting in DyDi-LiT that outperforms current SOTA models across multiple metrics.
Diffusion transformers (DiTs) have emerged as a powerful architecture for high-fidelity image generation, yet the quadratic cost of self-attention poses a major scalability bottleneck. To address this, linear attention mechanisms have been adopted to reduce computational cost; unfortunately, the resulting linear diffusion transformers (LiTs) models often come at the expense of generative performance, frequently producing over-smoothed attention weights that limit expressiveness. In this work, we introduce Dynamic Differential Linear Attention (DyDiLA), a novel linear attention formulation that enhances the effectiveness of LiTs by mitigating the oversmoothing issue and improving generation quality. Specifically, the novelty of DyDiLA lies in three key designs: (i) dynamic projection module, which facilitates the decoupling of token representations by learning with dynamically assigned knowledge; (ii) dynamic measure kernel, which provides a better similarity measurement to capture fine-grained semantic distinctions between tokens by dynamically assigning kernel functions for token processing; and (iii) token differential operator, which enables more robust query-to-key retrieval by calculating the differences between the tokens and their corresponding information redundancy produced by dynamic measure kernel. To capitalize on DyDiLA, we introduce a refined LiT, termed DyDi-LiT, that systematically incorporates our advancements. Extensive experiments show that DyDi-LiT consistently outperforms current state-of-the-art (SOTA) models across multiple metrics, underscoring its strong practical potential.