Attention in Diffusion Model: A Survey
It provides a comprehensive overview for researchers and practitioners working on diffusion models, highlighting current limitations and future directions, but is incremental as it synthesizes existing knowledge rather than introducing new methods.
This survey systematically analyzes the roles and design patterns of attention mechanisms in diffusion models, proposing a unified taxonomy to categorize modifications and examining their contributions to performance improvements across various applications.
Attention mechanisms have become a foundational component in diffusion models, significantly influencing their capacity across a wide range of generative and discriminative tasks. This paper presents a comprehensive survey of attention within diffusion models, systematically analysing its roles, design patterns, and operations across different modalities and tasks. We propose a unified taxonomy that categorises attention-related modifications into parts according to the structural components they affect, offering a clear lens through which to understand their functional diversity. In addition to reviewing architectural innovations, we examine how attention mechanisms contribute to performance improvements in diverse applications. We also identify current limitations and underexplored areas, and outline potential directions for future research. Our study provides valuable insights into the evolving landscape of diffusion models, with a particular focus on the integrative and ubiquitous role of attention.