Understanding Differential Transformer Unchains Pretrained Self-Attentions
This work addresses the challenge of leveraging Differential Transformer advantages without large-scale retraining, enabling more efficient use of pretrained models for NLP practitioners.
The paper tackled the problem of understanding and efficiently integrating Differential Transformer's benefits into pretrained language models, resulting in DEX, which improves pretrained LLMs across diverse benchmarks with significant performance gains using minimal adaptation data (<0.01%).
Differential Transformer has recently gained significant attention for its impressive empirical performance, often attributed to its ability to perform noise canceled attention. However, precisely how differential attention achieves its empirical benefits remains poorly understood. Moreover, Differential Transformer architecture demands large-scale training from scratch, hindering utilization of open pretrained weights. In this work, we conduct an in-depth investigation of Differential Transformer, uncovering three key factors behind its success: (1) enhanced expressivity via negative attention, (2) reduced redundancy among attention heads, and (3) improved learning dynamics. Based on these findings, we propose DEX, a novel method to efficiently integrate the advantages of differential attention into pretrained language models. By reusing the softmax attention scores and adding a lightweight differential operation on the output value matrix, DEX effectively incorporates the key advantages of differential attention while remaining lightweight in both training and inference. Evaluations confirm that DEX substantially improves the pretrained LLMs across diverse benchmarks, achieving significant performance gains with minimal adaptation data (< 0.01%).