FewMMBench: A Benchmark for Multimodal Few-Shot Learning
This work addresses the need for rigorous assessment of few-shot learning in multimodal AI, though it is incremental as it focuses on benchmarking rather than novel model development.
The authors tackled the challenge of evaluating few-shot learning capabilities in multimodal large language models by introducing FewMMBench, a comprehensive benchmark covering diverse multimodal tasks, and found that instruction-tuned models show strong zero-shot performance but gain little from demonstrations or chain-of-thought prompting.
As multimodal large language models (MLLMs) advance in handling interleaved image-text data, assessing their few-shot learning capabilities remains an open challenge. In this paper, we introduce FewMMBench, a comprehensive benchmark designed to evaluate MLLMs under few-shot conditions, with a focus on In-Context Learning (ICL) and Chain-of-Thought (CoT) prompting. Covering a diverse suite of multimodal understanding tasks, from attribute recognition to temporal reasoning, FewMMBench enables systematic analysis across task types, model families, and prompting strategies. We evaluate 26 open-weight MLLMs from six model families across zero-shot, few-shot, and CoT-augmented few-shot settings. Our findings reveal that instruction-tuned models exhibit strong zero-shot performance but benefit minimally, or even regress, with additional demonstrations or CoT reasoning. Retrieval-based demonstrations and increased context size also yield limited gains. These results highlight FewMMBench as a rigorous testbed for diagnosing and advancing few-shot capabilities in multimodal LLMs. The data is available at: https://huggingface.co/datasets/mustafaa/FewMMBench