DINT Transformer
This work addresses limitations in attention mechanisms for transformers, offering incremental improvements for tasks requiring global dependencies and numerical stability.
The paper tackled the problem of irrelevant context interference and lack of global context modeling in DIFF Transformer by proposing DINT Transformer, which incorporates a differential-integral mechanism and enforces row-normalized attention matrices, resulting in improved accuracy and robustness in applications like long-context language modeling and key information retrieval.
DIFF Transformer addresses the issue of irrelevant context interference by introducing a differential attention mechanism that enhances the robustness of local attention. However, it has two critical limitations: the lack of global context modeling, which is essential for identifying globally significant tokens, and numerical instability due to the absence of strict row normalization in the attention matrix. To overcome these challenges, we propose DINT Transformer, which extends DIFF Transformer by incorporating a differential-integral mechanism. By computing global importance scores and integrating them into the attention matrix, DINT Transformer improves its ability to capture global dependencies. Moreover, the unified parameter design enforces row-normalized attention matrices, improving numerical stability. Experimental results demonstrate that DINT Transformer excels in accuracy and robustness across various practical applications, such as long-context language modeling and key information retrieval. These results position DINT Transformer as a highly effective and promising architecture.