A multitask transformer to sign language translation using motion gesture primitives
This work addresses the communication gap for the deaf community by improving sign language translation, though it appears incremental as it builds on existing encoder-decoder and attention methods with specific enhancements.
The paper tackles the problem of automatic translation between sign language videos and natural text by introducing a multitask transformer architecture with gloss learning and dense motion representation, achieving state-of-the-art BLEU-4 scores of 72.64% and 14.64% on splits of the CoL-SLTD dataset and a competitive 11.58% on the RWTH-PHOENIX-Weather 2014 T dataset.
The absence of effective communication the deaf population represents the main social gap in this community. Furthermore, the sign language, main deaf communication tool, is unlettered, i.e., there is no formal written representation. In consequence, main challenge today is the automatic translation among spatiotemporal sign representation and natural text language. Recent approaches are based on encoder-decoder architectures, where the most relevant strategies integrate attention modules to enhance non-linear correspondences, besides, many of these approximations require complex training and architectural schemes to achieve reasonable predictions, because of the absence of intermediate text projections. However, they are still limited by the redundant background information of the video sequences. This work introduces a multitask transformer architecture that includes a gloss learning representation to achieve a more suitable translation. The proposed approach also includes a dense motion representation that enhances gestures and includes kinematic information, a key component in sign language. From this representation it is possible to avoid background information and exploit the geometry of the signs, in addition, it includes spatiotemporal representations that facilitate the alignment between gestures and glosses as an intermediate textual representation. The proposed approach outperforms the state-of-the-art evaluated on the CoL-SLTD dataset, achieving a BLEU-4 of 72,64% in split 1, and a BLEU-4 of 14,64% in split 2. Additionally, the strategy was validated on the RWTH-PHOENIX-Weather 2014 T dataset, achieving a competitive BLEU-4 of 11,58%.