EmoSign: A Multimodal Dataset for Understanding Emotions in American Sign Language
This addresses a critical gap in sign language research for improving communication in settings like healthcare or education, though it is incremental as it focuses on dataset creation.
The authors tackled the problem of understanding emotions in American Sign Language (ASL) by introducing EmoSign, the first dataset with sentiment and emotion labels for 200 ASL videos, annotated by Deaf signers, and including baseline models for classification.
Unlike spoken languages where the use of prosodic features to convey emotion is well studied, indicators of emotion in sign language remain poorly understood, creating communication barriers in critical settings. Sign languages present unique challenges as facial expressions and hand movements simultaneously serve both grammatical and emotional functions. To address this gap, we introduce EmoSign, the first sign video dataset containing sentiment and emotion labels for 200 American Sign Language (ASL) videos. We also collect open-ended descriptions of emotion cues. Annotations were done by 3 Deaf ASL signers with professional interpretation experience. Alongside the annotations, we include baseline models for sentiment and emotion classification. This dataset not only addresses a critical gap in existing sign language research but also establishes a new benchmark for understanding model capabilities in multimodal emotion recognition for sign languages. The dataset is made available at https://huggingface.co/datasets/catfang/emosign.