Leveraging Textual-Cues for Enhancing Multimodal Sentiment Analysis by Object Recognition
This work addresses challenges in multimodal sentiment analysis for applications like social media analysis, but it appears incremental as it builds on existing object recognition methods.
The paper tackled multimodal sentiment analysis by combining image and text data using object recognition to extract object names and integrate them with text, resulting in improved sentiment analysis performance compared to individual modality analysis.
Multimodal sentiment analysis, which includes both image and text data, presents several challenges due to the dissimilarities in the modalities of text and image, the ambiguity of sentiment, and the complexities of contextual meaning. In this work, we experiment with finding the sentiments of image and text data, individually and in combination, on two datasets. Part of the approach introduces the novel `Textual-Cues for Enhancing Multimodal Sentiment Analysis' (TEMSA) based on object recognition methods to address the difficulties in multimodal sentiment analysis. Specifically, we extract the names of all objects detected in an image and combine them with associated text; we call this combination of text and image data TEMS. Our results demonstrate that only TEMS improves the results when considering all the object names for the overall sentiment of multimodal data compared to individual analysis. This research contributes to advancing multimodal sentiment analysis and offers insights into the efficacy of TEMSA in combining image and text data for multimodal sentiment analysis.