Multi-layer Learnable Attention Mask for Multimodal Tasks
This is an incremental improvement for multimodal AI, addressing computational challenges in tasks like movie understanding.
The paper tackled the inefficiency of Self-Attention in multimodal tasks by introducing a Learnable Attention Mask (LAM) to regulate attention and prioritize critical tokens, resulting in enhanced model performance and reduced computations across datasets like MADv2 and ImageNet 1K.
While the Self-Attention mechanism in the Transformer model has proven to be effective in many domains, we observe that it is less effective in more diverse settings (e.g. multimodality) due to the varying granularity of each token and the high computational demands of lengthy sequences. To address the challenges, we introduce the Learnable Attention Mask (LAM), strategically designed to globally regulate attention maps and prioritize critical tokens within the sequence. Leveraging the Self-Attention module in a BERT-like transformer network, our approach adeptly captures associations between tokens. The extension of the LAM to a multi-layer version accommodates the varied information aspects embedded at each layer of the Transformer network. Comprehensive experimental validation on various datasets, such as MADv2, QVHighlights, ImageNet 1K, and MSRVTT, demonstrates the efficacy of the LAM, exemplifying its ability to enhance model performance while mitigating redundant computations. This pioneering approach presents a significant advancement in enhancing the understanding of complex scenarios, such as in movie understanding.