Are Large Language Models Chronically Online Surfers? A Dataset for Chinese Internet Meme Explanation
This work addresses the challenge of computational meme understanding for researchers and developers, but it is incremental as it focuses on dataset creation and evaluation without proposing new methods.
The paper tackles the problem of whether large language models (LLMs) understand Chinese internet memes by introducing the CHIME dataset, and finds that while LLMs can explain some memes, their performance declines significantly for nuanced types and they struggle with accurate origins, with performance below human levels on multiple-choice tasks.
Large language models (LLMs) are trained on vast amounts of text from the Internet, but do they truly understand the viral content that rapidly spreads online -- commonly known as memes? In this paper, we introduce CHIME, a dataset for CHinese Internet Meme Explanation. The dataset comprises popular phrase-based memes from the Chinese Internet, annotated with detailed information on their meaning, origin, example sentences, types, etc. To evaluate whether LLMs understand these memes, we designed two tasks. In the first task, we assessed the models' ability to explain a given meme, identify its origin, and generate appropriate example sentences. The results show that while LLMs can explain the meanings of some memes, their performance declines significantly for culturally and linguistically nuanced meme types. Additionally, they consistently struggle to provide accurate origins for the memes. In the second task, we created a set of multiple-choice questions (MCQs) requiring LLMs to select the most appropriate meme to fill in a blank within a contextual sentence. While the evaluated models were able to provide correct answers, their performance remains noticeably below human levels. We have made CHIME public and hope it will facilitate future research on computational meme understanding.