Lexical Retrieval Hypothesis in Multimodal Context
This provides a new resource for multimodal communication studies, but the findings are incremental as they partially confirm and extend existing hypotheses.
The authors built the first Multimodal Corpus for Languages in Taiwan (MultiMoco) and used it to test the Lexical Retrieval Hypothesis, finding that hand gestures sometimes facilitate lexical retrieval but also serve for information emphasis.
Multimodal corpora have become an essential language resource for language science and grounded natural language processing (NLP) systems due to the growing need to understand and interpret human communication across various channels. In this paper, we first present our efforts in building the first Multimodal Corpus for Languages in Taiwan (MultiMoco). Based on the corpus, we conduct a case study investigating the Lexical Retrieval Hypothesis (LRH), specifically examining whether the hand gestures co-occurring with speech constants facilitate lexical retrieval or serve other discourse functions. With detailed annotations on eight parliamentary interpellations in Taiwan Mandarin, we explore the co-occurrence between speech constants and non-verbal features (i.e., head movement, face movement, hand gesture, and function of hand gesture). Our findings suggest that while hand gestures do serve as facilitators for lexical retrieval in some cases, they also serve the purpose of information emphasis. This study highlights the potential of the MultiMoco Corpus to provide an important resource for in-depth analysis and further research in multimodal communication studies.