Beyond the Literal: Decomposing Pragmatic Intent in Multimodal Meme Understanding
For researchers and practitioners in multimodal understanding, this work addresses the critical bottleneck of literal-pragmatic entanglement in LVLMs, offering a novel decomposition approach that significantly improves pragmatic interpretation.
The paper tackles the problem of Large Vision Language Models (LVLMs) describing literal content instead of pragmatic intent in multimodal memes. The proposed Intent Projection framework separates literal and pragmatic signals, achieving consistent improvements over open-source baselines across six benchmarks, with largest gains on high-divergence posts.
When asked what a meme or sarcastic post means, Large Vision Language Models (LVLMs) tend to describe what the image shows rather than what the author is trying to communicate. Standard instruction tuning entangles a post's literal content with its pragmatic meaning, letting surface-level details contaminate the final response. We reframe meme understanding as a problem of literal-pragmatic decomposition and propose \textbf{Intent Projection}, a framework that separates the two signals at the representation, output, and objective levels within a single LVLM backbone. At the representation level, an orthogonal projection module removes dominant unimodal directions from the fused image-text representation, retaining only the pragmatic residual, while a surface-real affect classifier anchors the decoder with a discrete tag that names the polarity gap. At the output level, the model externalizes a structured reasoning chain, and at the objective level a contrastive reward explicitly penalizes answers that restate the literal description. Across six multimodal benchmarks, Intent Projection consistently outperforms open-source baselines and narrows the gap to proprietary models, with the largest gains on high-divergence posts where literal collapse is most damaging.