Hierachical Delta-Attention Method for Multimodal Fusion
This work provides an incremental improvement for researchers in multimodal fusion by offering a more parameter-efficient approach to integrate local nuances and global context from different modalities.
The paper introduces a hierarchical delta-attention method for multimodal fusion, addressing the loss of contextual information when analyzing single modalities in vision and linguistics. The method preserves long-range dependencies within and across modalities and focuses on local differences using delta-attention, achieving competitive accuracy close to state-of-the-art with almost half the number of parameters.
In vision and linguistics; the main input modalities are facial expressions, speech patterns, and the words uttered. The issue with analysis of any one mode of expression (Visual, Verbal or Vocal) is that lot of contextual information can get lost. This asks researchers to inspect multiple modalities to get a thorough understanding of the cross-modal dependencies and temporal context of the situation to analyze the expression. This work attempts at preserving the long-range dependencies within and across different modalities, which would be bottle-necked by the use of recurrent networks and adds the concept of delta-attention to focus on local differences per modality to capture the idiosyncrasy of different people. We explore a cross-attention fusion technique to get the global view of the emotion expressed through these delta-self-attended modalities, in order to fuse all the local nuances and global context together. The addition of attention is new to the multi-modal fusion field and currently being scrutinized for on what stage the attention mechanism should be used, this work achieves competitive accuracy for overall and per-class classification which is close to the current state-of-the-art with almost half number of parameters.