MMVA: Multimodal Matching Based on Valence and Arousal across Images, Music, and Musical Captions
This work addresses the challenge of multimodal emotional matching for applications in AI and human-computer interaction, representing an incremental advancement with a new dataset and framework.
The paper tackles the problem of capturing emotional content across images, music, and musical captions by introducing MMVA, a tri-modal encoder framework based on valence and arousal, which achieves state-of-the-art performance in valence-arousal prediction tasks and shows efficacy in zeroshot tasks.
We introduce Multimodal Matching based on Valence and Arousal (MMVA), a tri-modal encoder framework designed to capture emotional content across images, music, and musical captions. To support this framework, we expand the Image-Music-Emotion-Matching-Net (IMEMNet) dataset, creating IMEMNet-C which includes 24,756 images and 25,944 music clips with corresponding musical captions. We employ multimodal matching scores based on the continuous valence (emotional positivity) and arousal (emotional intensity) values. This continuous matching score allows for random sampling of image-music pairs during training by computing similarity scores from the valence-arousal values across different modalities. Consequently, the proposed approach achieves state-of-the-art performance in valence-arousal prediction tasks. Furthermore, the framework demonstrates its efficacy in various zeroshot tasks, highlighting the potential of valence and arousal predictions in downstream applications.