Attention Sinks in Diffusion Language Models
This provides insights into the inner workings of diffusion-based language models, highlighting differences from autoregressive models, but is incremental as it builds on prior observations of attention sinks in transformers.
The paper tackled the problem of understanding internal mechanisms in Masked Diffusion Language Models (DLMs) by analyzing attention patterns, specifically the attention sinking phenomenon, and found that DLMs exhibit dynamic sink positions and robustness to sink removal with only minor performance degradation.
Masked Diffusion Language Models (DLMs) have recently emerged as a promising alternative to traditional Autoregressive Models (ARMs). DLMs employ transformer encoders with bidirectional attention, enabling parallel token generation while maintaining competitive performance. Although their efficiency and effectiveness have been extensively studied, the internal mechanisms that govern DLMs remain largely unexplored. In this work, we conduct an empirical analysis of DLM attention patterns, focusing on the attention sinking phenomenon, an effect previously observed in various transformer-based architectures. Our findings reveal that DLMs also exhibit attention sinks, but with distinct characteristics. First, unlike in ARMs, the sink positions in DLMs tend to shift throughout the generation process, displaying a dynamic behaviour. Second, while ARMs are highly sensitive to the removal of attention sinks, DLMs remain robust: masking sinks leads to only a minor degradation in performance. These results provide new insights into the inner workings of diffusion-based language models and highlight fundamental differences in how they allocate and utilize attention compared to autoregressive models.