Addressing the Blind Spots in Spoken Language Processing
It addresses a critical gap in NLP for applications requiring holistic human communication analysis, though it is incremental as it builds on existing sign language processing methods.
This paper tackles the problem of incomplete spoken language understanding in NLP by highlighting the importance of non-verbal cues like gestures and facial expressions, proposing universal automatic gesture segmentation and transcription models to transcribe these cues into text and bridge blind spots.
This paper explores the critical but often overlooked role of non-verbal cues, including co-speech gestures and facial expressions, in human communication and their implications for Natural Language Processing (NLP). We argue that understanding human communication requires a more holistic approach that goes beyond textual or spoken words to include non-verbal elements. Borrowing from advances in sign language processing, we propose the development of universal automatic gesture segmentation and transcription models to transcribe these non-verbal cues into textual form. Such a methodology aims to bridge the blind spots in spoken language understanding, enhancing the scope and applicability of NLP models. Through motivating examples, we demonstrate the limitations of relying solely on text-based models. We propose a computationally efficient and flexible approach for incorporating non-verbal cues, which can seamlessly integrate with existing NLP pipelines. We conclude by calling upon the research community to contribute to the development of universal transcription methods and to validate their effectiveness in capturing the complexities of real-world, multi-modal interactions.