Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmark
This work addresses a gap in multimodal language analysis for researchers by providing a benchmark to assess model limitations, though it is incremental as it focuses on evaluation rather than new methods.
The paper tackles the problem of evaluating multimodal large language models' ability to understand cognitive-level semantics in human conversations by introducing MMLA, a comprehensive benchmark with over 61K multimodal utterances across six dimensions, and finds that even fine-tuned models achieve only about 60%~70% accuracy.
Multimodal language analysis is a rapidly evolving field that leverages multiple modalities to enhance the understanding of high-level semantics underlying human conversational utterances. Despite its significance, little research has investigated the capability of multimodal large language models (MLLMs) to comprehend cognitive-level semantics. In this paper, we introduce MMLA, a comprehensive benchmark specifically designed to address this gap. MMLA comprises over 61K multimodal utterances drawn from both staged and real-world scenarios, covering six core dimensions of multimodal semantics: intent, emotion, dialogue act, sentiment, speaking style, and communication behavior. We evaluate eight mainstream branches of LLMs and MLLMs using three methods: zero-shot inference, supervised fine-tuning, and instruction tuning. Extensive experiments reveal that even fine-tuned models achieve only about 60%~70% accuracy, underscoring the limitations of current MLLMs in understanding complex human language. We believe that MMLA will serve as a solid foundation for exploring the potential of large language models in multimodal language analysis and provide valuable resources to advance this field. The datasets and code are open-sourced at https://github.com/thuiar/MMLA.