Multimodal Conversation Structure Understanding
This work addresses the underexplored ability of LLMs to understand conversational structure in multimodal settings, which is incremental as it builds on existing models and datasets for improved evaluation.
The paper tackled the problem of understanding fine-grained conversational structure in multimodal, multi-party settings by introducing tasks for role attribution and conversation threading, and created a human-annotated dataset with thousands of annotations; evaluation showed that multimodal conversational structure understanding is challenging, with the best audio-visual LLM outperforming vision-language models but performance dropping significantly with anonymized participants and more participants negatively affecting role-attribution.
Conversations are usually structured by roles -- who is speaking, who's being addressed, and who's listening -- and unfold in threads that break with changes in speaker floor or topical focus. While large language models (LLMs) have shown incredible capabilities in dialogue and reasoning, their ability to understand fine-grained conversational structure, especially in multi-modal, multi-party settings, remains underexplored. To address this gap, we introduce a suite of tasks focused on conversational role attribution (speaker, addressees, side-participants) and conversation threading (utterance linking and clustering), drawing on conversation analysis and sociolinguistics. To support those tasks, we present a human annotated dataset of 4,398 annotations for speakers and reply-to relationship, 5,755 addressees, and 3,142 side-participants. We evaluate popular audio-visual LLMs and vision-language models on our dataset, and our experimental results suggest that multimodal conversational structure understanding remains challenging. The most performant audio-visual LLM outperforms all vision-language models across all metrics, especially in speaker and addressee recognition. However, its performance drops significantly when conversation participants are anonymized. The number of conversation participants in a clip is the strongest negative predictor of role-attribution performance, while acoustic clarity (measured by pitch and spectral centroid) and detected face coverage yield positive associations. We hope this work lays the groundwork for future evaluation and development of multimodal LLMs that can reason more effectively about conversation structure.