ZoDIAC: Zoneout Dropout Injection Attention Calculation
This work addresses a specific bottleneck in transformer models for tasks like image captioning, offering a drop-in replacement for attention components with incremental improvements.
The paper tackles the lack of explicit refinement and intensification of attention values in transformer self-attention by introducing ZoDIAC, a novel mechanism that refines attention using GELU and dropout and intensifies it with a zoneup process, achieving statistically significant higher scores across all image captioning metrics on the MS-COCO dataset.
In the past few years the transformer model has been utilized for a variety of tasks such as image captioning, image classification natural language generation, and natural language understanding. As a key component of the transformer model, self-attention calculates the attention values by mapping the relationships among the head elements of the source and target sequence, yet there is no explicit mechanism to refine and intensify the attention values with respect to the context of the input and target sequences. Based on this intuition, we introduce a novel refine and intensify attention mechanism that is called Zoneup Dropout Injection Attention Calculation (ZoDIAC), in which the intensities of attention values in the elements of the input source and target sequences are first refined using GELU and dropout and then intensified using a proposed zoneup process which includes the injection of a learned scalar factor. Our extensive experiments show that ZoDIAC achieves statistically significant higher scores under all image captioning metrics using various feature extractors in comparison to the conventional self-attention module in the transformer model on the MS-COCO dataset. Our proposed ZoDIAC attention modules can be used as a drop-in replacement for the attention components in all transformer models. The code for our experiments is publicly available at: https://github.com/zanyarz/zodiac