Why self-attention is Natural for Sequence-to-Sequence Problems? A Perspective from Symmetries
This provides a theoretical foundation for self-attention in seq2seq problems, which is incremental as it explains existing methods rather than introducing new ones.
The paper tackles the problem of justifying self-attention in sequence-to-sequence tasks by showing that orthogonal equivariance in embedding spaces naturally leads to structures similar to self-attention, demonstrating it as the right representation for many seq2seq functions.
In this paper, we show that structures similar to self-attention are natural to learn many sequence-to-sequence problems from the perspective of symmetry. Inspired by language processing applications, we study the orthogonal equivariance of seq2seq functions with knowledge, which are functions taking two inputs -- an input sequence and a ``knowledge'' -- and outputting another sequence. The knowledge consists of a set of vectors in the same embedding space as the input sequence, containing the information of the language used to process the input sequence. We show that orthogonal equivariance in the embedding space is natural for seq2seq functions with knowledge, and under such equivariance the function must take the form close to the self-attention. This shows that network structures similar to self-attention are the right structures to represent the target function of many seq2seq problems. The representation can be further refined if a ``finite information principle'' is considered, or a permutation equivariance holds for the elements of the input sequence.