A Self-Adjusting Fusion Representation Learning Model for Unaligned Text-Audio Sequences
This addresses the problem of multimodal sentiment analysis for researchers and practitioners by enhancing fusion representation learning, though it appears incremental as it builds on existing methods for unaligned sequences.
The paper tackles the challenge of learning fusion representations from unaligned text-audio sequences in multimodal sentiment analysis by proposing a Self-Adjusting Fusion Representation Learning Model (SA-FRLM), which significantly improves performance on datasets like CMU-MOSI and CMU-MOSEI.
Inter-modal interaction plays an indispensable role in multimodal sentiment analysis. Due to different modalities sequences are usually non-alignment, how to integrate relevant information of each modality to learn fusion representations has been one of the central challenges in multimodal learning. In this paper, a Self-Adjusting Fusion Representation Learning Model (SA-FRLM) is proposed to learn robust crossmodal fusion representations directly from the unaligned text and audio sequences. Different from previous works, our model not only makes full use of the interaction between different modalities but also maximizes the protection of the unimodal characteristics. Specifically, we first employ a crossmodal alignment module to project different modalities features to the same dimension. The crossmodal collaboration attention is then adopted to model the inter-modal interaction between text and audio sequences and initialize the fusion representations. After that, as the core unit of the SA-FRLM, the crossmodal adjustment transformer is proposed to protect original unimodal characteristics. It can dynamically adapt the fusion representations by using single modal streams. We evaluate our approach on the public multimodal sentiment analysis datasets CMU-MOSI and CMU-MOSEI. The experiment results show that our model has significantly improved the performance of all the metrics on the unaligned text-audio sequences.