Revisiting Linformer with a modified self-attention with linear complexity
This work aims to improve the efficiency of Transformer models for researchers and practitioners by reducing the computational bottleneck of self-attention, offering an incremental improvement over existing linear-complexity methods.
This paper addresses the quadratic time and space complexity of self-attention in Transformer models, which makes them costly to train and deploy. The author proposes a modified self-attention mechanism that achieves linear complexity in time and space, and unlike previous work like Linformer, its performance is independent of a projection mapping dimension.
Although Transformer models such as Google's BERT and OpenAI's GPT-3 are successful in many natural language processing tasks, training and deploying these models are costly and inefficient.Even if pre-trained models are used, deploying these models still remained a challenge due to their large size. Apart from deployment, these models take higher time during inference restricting user-friendliness. The main bottleneck is self-attention which uses quadratic time and space with respect to the sequence length. In order to reduce the quadratic time complexity of the self-attention mechanism, Linformer by Facebook's AI research team was introduced where they showed that the self-attention mechanism can be approximated by a low-rank matrix and exploiting this finding, a new method for self-attention with linear time and space complexity was proposed by them. In the Linformer, the time complexity depends on the projection mapping dimension which acts as a hyperparameter and affects the performance of the model, tuning this hyperparameter can be time-consuming. In this paper, I proposed an alternative method for self-attention with linear complexity in time and space and is independent of the projection mapping dimension. Since this method works for long sequences this can be used for images as well as audios.