DeepCrossAttention: Supercharging Transformer Residual Connections
This is an incremental improvement for transformer-based models, potentially benefiting applications in language modeling and other domains by enhancing efficiency.
The paper tackles the problem of traditional residual connections diluting information in transformers by introducing DeepCrossAttention (DCA), which uses learnable weights and depth-wise cross-attention to dynamically combine layer outputs, resulting in up to 3x faster training for the same model quality with negligible parameter increase.
Transformer networks have achieved remarkable success across diverse domains, leveraging a variety of architectural innovations, including residual connections. However, traditional residual connections, which simply sum the outputs of previous layers, can dilute crucial information. This work introduces DeepCrossAttention (DCA), an approach that enhances residual learning in transformers. DCA employs learnable, input-dependent weights to dynamically combine layer outputs, enabling the model to selectively focus on the most relevant information in any of the previous layers. Furthermore, DCA incorporates depth-wise cross-attention, allowing for richer interactions between layers at different depths. Our language modeling experiments show that DCA achieves improved perplexity for a given training time. Moreover, DCA obtains the same model quality up to 3x faster while adding a negligible number of parameters. Theoretical analysis confirms that DCA provides an improved trade-off between accuracy and model size when the ratio of collective layer ranks to the ambient dimension falls below a critical threshold.