Value-aware Approximate Attention
This work addresses a key bottleneck in scaling Transformers for long sequences, offering a novel approach to improve efficiency and accuracy in natural language processing tasks.
The paper tackles the problem of approximating attention in Transformers by arguing that existing methods ignore the role of value vectors, proposing a value-aware objective that theoretically and empirically outperforms value-ignoring approximations in language modeling, with substantial performance gains.
Following the success of dot-product attention in Transformers, numerous approximations have been recently proposed to address its quadratic complexity with respect to the input length. However, all approximations thus far have ignored the contribution of the $\textit{value vectors}$ to the quality of approximation. In this work, we argue that research efforts should be directed towards approximating the true output of the attention sub-layer, which includes the value vectors. We propose a value-aware objective, and show theoretically and empirically that an optimal approximation of a value-aware objective substantially outperforms an optimal approximation that ignores values, in the context of language modeling. Moreover, we show that the choice of kernel function for computing attention similarity can substantially affect the quality of sparse approximations, where kernel functions that are less skewed are more affected by the value vectors.