Multimodal UNcommonsense: From Odd to Ordinary and Ordinary to Odd
This work addresses the problem of evaluating and improving visual-language models' robustness in non-prototypical scenarios for AI researchers and practitioners, though it is incremental as it builds on existing ICL methods.
The paper tackles the challenge of commonsense reasoning in multimodal contexts by introducing the Multimodal UNcommonsense (MUN) benchmark to evaluate models on scenarios with surprising or unlikely outcomes, and proposes a retrieval-based in-context learning (R-ICL) framework that improves over baseline methods by 8.3% on average.
Commonsense reasoning in multimodal contexts remains a foundational challenge in artificial intelligence. We introduce Multimodal UNcommonsense(MUN), a benchmark designed to evaluate models' ability to handle scenarios that deviate from typical visual or contextual expectations. MUN pairs visual scenes with surprising or unlikely outcomes described in natural language, prompting models to either rationalize seemingly odd images using everyday logic or uncover unexpected interpretations in ordinary scenes. To support this task, we propose a retrieval-based in-context learning (R-ICL) framework that transfers reasoning capabilities from larger models to smaller ones without additional training. Leveraging a novel Multimodal Ensemble Retriever (MER), our method identifies semantically relevant exemplars even when image and text pairs are deliberately discordant. Experiments show an average improvement of 8.3% over baseline ICL methods, highlighting the effectiveness of R-ICL in low-frequency, atypical settings. MUN opens new directions for evaluating and improving visual-language models' robustness and adaptability in real-world, culturally diverse, and non-prototypical scenarios.