The Hateful Memes Challenge Next Move
This work addresses the challenge of multimodal classification for hateful memes, but it is incremental as it builds on existing frameworks without yielding new gains.
The study tackled the problem of classifying hateful memes by attempting to generate more labeled data using semi-supervised learning with pseudo-labels from an additional dataset, but found that human intervention was needed and no performance improvement was achieved.
State-of-the-art image and text classification models, such as Convolutional Neural Networks and Transformers, have long been able to classify their respective unimodal reasoning satisfactorily with accuracy close to or exceeding human accuracy. However, images embedded with text, such as hateful memes, are hard to classify using unimodal reasoning when difficult examples, such as benign confounders, are incorporated into the data set. We attempt to generate more labeled memes in addition to the Hateful Memes data set from Facebook AI, based on the framework of a winning team from the Hateful Meme Challenge. To increase the number of labeled memes, we explore semi-supervised learning using pseudo-labels for newly introduced, unlabeled memes gathered from the Memotion Dataset 7K. We find that the semi-supervised learning task on unlabeled data required human intervention and filtering and that adding a limited amount of new data yields no extra classification performance.