Shared DIFF Transformer
This work addresses efficiency and parameter optimization in Transformer architectures for researchers and practitioners in NLP and sequence modeling, representing an incremental improvement over existing differential attention methods.
The authors tackled the parameter redundancy and suboptimal information utilization in DIFF Transformer's differential attention mechanism by proposing Shared DIFF Transformer, which introduces a shared base matrix with low-rank updates to model global patterns and enhance task-specific flexibility. This approach reduces parameters while maintaining noise suppression, achieving better performance in long-sequence modeling, key information retrieval, and in-context learning compared to DIFF Transformer.
DIFF Transformer improves attention allocation by enhancing focus on relevant context while suppressing noise. It introduces a differential attention mechanism that calculates the difference between two independently generated attention distributions, effectively reducing noise and promoting sparse attention patterns. However, the independent signal generation in DIFF Transformer results in parameter redundancy and suboptimal utilization of information. In this work, we propose Shared DIFF Transformer, which draws on the idea of a differential amplifier by introducing a shared base matrix to model global patterns and incorporating low-rank updates to enhance task-specific flexibility. This design significantly reduces parameter redundancy, improves efficiency, and retains strong noise suppression capabilities. Experimental results show that, compared to DIFF Transformer, our method achieves better performance in tasks such as long-sequence modeling, key information retrieval, and in-context learning. Our work provides a novel and efficient approach to optimizing differential attention mechanisms and advancing robust Transformer architectures.