A Dataset and Benchmarks for Multimedia Social Analysis
This provides a resource for researchers in AI and computer vision working on multi-modal tasks, but it is incremental as it focuses on dataset creation without introducing new methods.
The authors tackled the need for multi-modality learning by creating a large dataset from social media with paired images/videos and text, containing 677k posts and millions of multimedia elements, to improve tasks like image captioning and sentiment analysis.
We present a new publicly available dataset with the goal of advancing multi-modality learning by offering vision and language data within the same context. This is achieved by obtaining data from a social media website with posts containing multiple paired images/videos and text, along with comment trees containing images/videos and/or text. With a total of 677k posts, 2.9 million post images, 488k post videos, 1.4 million comment images, 4.6 million comment videos, and 96.9 million comments, data from different modalities can be jointly used to improve performances for a variety of tasks such as image captioning, image classification, next frame prediction, sentiment analysis, and language modeling. We present a wide range of statistics for our dataset. Finally, we provide baseline performance analysis for one of the regression tasks using pre-trained models and several fully connected networks.