Quaternion Self-Attention with Shared Scores
For practitioners using quaternion neural networks, this work offers a computationally cheaper self-attention variant with minimal quality loss.
The paper proposes a shared-score quaternion self-attention mechanism that reduces score-computation multiplications by 75% and softmax operations from four to one, achieving up to 44.3% GPU and 58.1% CPU inference time reduction in speech enhancement while maintaining quality across vision and NLP tasks.
Quaternion neural networks are parameter-efficient and model multidimensional dependencies by representing four related features as a single entity. However, existing quaternion self-attention computes component-wise scores and applies independent softmax operations to each component, which increases the computational cost and allows attention distributions to diverge across components. We propose a shared-score quaternion self-attention mechanism that computes a single real-valued score using the quaternion inner product and applies a shared attention distribution across all components. This reduces score-computation multiplications by 75% and the number of softmax operations from four to one. We prove that, when queries and keys are produced by quaternion linear projections that induce component pre-mixing, the component-wise and shared scores lie in the same interaction subspace, indicating that independent component-wise attention primarily re-parameterizes the same interactions rather than expanding the feature interaction space. In speech enhancement, our method reduces inference time by up to 44.3% on a GPU and 58.1% on a CPU while maintaining quality, with consistent trends across vision and natural language processing.