MTAG: Modal-Temporal Attention Graph for Unaligned Human Multimodal Language Sequences
This addresses the challenge of modeling complex multimodal interactions for applications like sentiment analysis, though it appears incremental as it builds on existing graph-based methods.
The paper tackled the problem of learning from unaligned multimodal human communication sequences by proposing MTAG, a graph-based neural model that converts data into a heterogeneous graph and uses novel fusion and pruning techniques, achieving state-of-the-art performance on sentiment analysis and emotion recognition benchmarks with fewer parameters.
Human communication is multimodal in nature; it is through multiple modalities such as language, voice, and facial expressions, that opinions and emotions are expressed. Data in this domain exhibits complex multi-relational and temporal interactions. Learning from this data is a fundamentally challenging research problem. In this paper, we propose Modal-Temporal Attention Graph (MTAG). MTAG is an interpretable graph-based neural model that provides a suitable framework for analyzing multimodal sequential data. We first introduce a procedure to convert unaligned multimodal sequence data into a graph with heterogeneous nodes and edges that captures the rich interactions across modalities and through time. Then, a novel graph fusion operation, called MTAG fusion, along with a dynamic pruning and read-out technique, is designed to efficiently process this modal-temporal graph and capture various interactions. By learning to focus only on the important interactions within the graph, MTAG achieves state-of-the-art performance on multimodal sentiment analysis and emotion recognition benchmarks, while utilizing significantly fewer model parameters.