Beyond Parallelism: Synergistic Computational Graph Effects in Multi-Head Attention
This work addresses a foundational problem in deep learning for researchers and practitioners by providing theoretical insights into multi-head attention in Transformers, though it appears incremental as it builds on existing frameworks.
The paper tackled the theoretical advantages of multi-head attention beyond parallelism by reframing it as synergistic computational graphs, showing that it can enhance information propagation with faster mixing times and minimax fidelity under head-diversity conditions, and empirically verified these effects on sequence manipulation tasks.
Multi-head attention powers Transformer networks, the primary deep learning architecture behind the success of large language models (LLMs). Yet, the theoretical advantages of multi-head versus single-head attention, beyond mere parallel processing, remain underexplored. In this paper, we reframe multi-head attention as a system of potentially synergistic computational graphs, where each head functions as a feedforward directed acyclic graph (DAG) with a common sink state. We provide intuition and preliminary theoretical analysis of mixing time and minimax fidelity in this framework. Our results show that multi-head attention can synergistically enhance information propagation, yielding faster mixing times and minimax fidelity amplification under specific head-diversity conditions. Finally, we train single-head and multi-head Transformers, each with the same total number of parameters, on sequence manipulation tasks and empirically verify the predicted effects. The code is available at https://github.com/haitzsaezdeocariz/beyondparallelism.