Orthogonal Self-Attention
This addresses a critical training issue for researchers developing efficient Transformer architectures, though it is an incremental improvement focused on a specific bottleneck.
The paper tackles the instability of Softmax Self-Attention in skipless Transformers by introducing Orthogonal Self-Attention, which ensures well-conditioned Jacobians and scales linearly with sequence length, enabling stable training without skip connections or normalization layers.
Softmax Self-Attention (SSA) is a key component of Transformer architectures. However, when utilised within skipless architectures, which aim to improve representation learning, recent work has highlighted the inherent instability of SSA due to inducing rank collapse and poorly-conditioned Jacobians. In this work, we design a novel attention mechanism: Orthogonal Self-Attention (OSA), which aims to bypass these issues with SSA, in order to allow for (non-causal) Transformers without skip connections and normalisation layers to be more easily trained. In particular, OSA parametrises the attention matrix to be orthogonal via mapping a skew-symmetric matrix, formed from query-key values, through the matrix exponential. We show that this can be practically implemented, by exploiting the low-rank structure of our query-key values, resulting in the computational complexity and memory cost of OSA scaling linearly with sequence length. Furthermore, we derive an initialisation scheme for which we prove ensures that the Jacobian of OSA is well-conditioned.