Cross-Attention Based Influence Model for Manual and Nonmanual Sign Language Analysis
This work addresses the challenge of incomplete sign language translation for deaf and hearing-impaired communities by integrating non-manual features, though it is incremental as it builds on existing neural machine translation methods.
The authors tackled the problem of understanding American Sign Language (ASL) by incorporating both manual and non-manual markers, such as facial expressions, into a translation model, and found that their cross-attention mechanism allowed analysis of the influence of facial markers compared to body and hand features.
Both manual (relating to the use of hands) and non-manual markers (NMM), such as facial expressions or mouthing cues, are important for providing the complete meaning of phrases in American Sign Language (ASL). Efforts have been made in advancing sign language to spoken/written language understanding, but most of these have primarily focused on manual features only. In this work, using advanced neural machine translation methods, we examine and report on the extent to which facial expressions contribute to understanding sign language phrases. We present a sign language translation architecture consisting of two-stream encoders, with one encoder handling the face and the other handling the upper body (with hands). We propose a new parallel cross-attention decoding mechanism that is useful for quantifying the influence of each input modality on the output. The two streams from the encoder are directed simultaneously to different attention stacks in the decoder. Examining the properties of the parallel cross-attention weights allows us to analyze the importance of facial markers compared to body and hand features during a translating task.