DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averaging
This is an incremental improvement for transformer-based models in domains like NLP, speech, and image processing, offering better performance without increasing model size.
The paper tackles the problem of inefficient information flow in transformers by proposing DenseFormer, which uses depth-weighted averaging to enhance data efficiency, achieving the same perplexity as deeper models with fewer parameters and improving memory efficiency and inference time.
The transformer architecture by Vaswani et al. (2017) is now ubiquitous across application domains, from natural language processing to speech processing and image understanding. We propose DenseFormer, a simple modification to the standard architecture that improves the perplexity of the model without increasing its size -- adding a few thousand parameters for large-scale models in the 100B parameters range. Our approach relies on an additional averaging step after each transformer block, which computes a weighted average of current and past representations -- we refer to this operation as Depth-Weighted-Average (DWA). The learned DWA weights exhibit coherent patterns of information flow, revealing the strong and structured reuse of activations from distant layers. Experiments demonstrate that DenseFormer is more data efficient, reaching the same perplexity of much deeper transformer models, and that for the same perplexity, these new models outperform transformer baselines in terms of memory efficiency and inference time.