Adaptive Captioning with Emotional Cues: Supporting DHH and Neurodivergent Learners in STEM
This addresses accessibility challenges for DHH and neurodivergent learners in STEM, but it is an incremental design-oriented exploration.
The paper tackled the problem of real-time captioning omitting emotional and non-verbal cues for Deaf and Hard of Hearing (DHH) and neurodivergent learners in STEM education, finding that certain prototypes reduced cognitive load and improved comprehension scores compared to traditional captions.
Real-time captioning is vital for Deaf and Hard of Hearing (DHH) and neurodivergent learners (e.g., those with ADHD), yet it often omits emotional and non-verbal cues essential for comprehension. This omission is particularly consequential in STEM education, where cognitively demanding material can exacerbate the challenges faced by caption users across diverse ability profiles. In this paper, we present a design-oriented exploration of four captioning prototypes that embed emotional and multimodal cues, including facial expressions, body gestures, keyword highlighting, and emoji. Across a pilot and a main study with 24 participants, we found that certain prototypes reduced self-reported cognitive load and improved comprehension scores compared to traditional captions. Qualitative feedback reveals the importance of customizable caption features to accommodate neurodivergent users' preferences (e.g., ADHD or different levels of comfort with emojis). Our findings contribute to ongoing conversations in accessible technology research about how best to integrate emotional cues into captions in a way that is both usable and beneficial for a wide range of learners.