Explore More Guidance: A Task-aware Instruction Network for Sign Language Translation Enhanced with Data Augmentation
This work addresses sign language translation, a domain-specific problem often overlooked compared to recognition, with incremental improvements over existing methods.
The paper tackles sign language translation by proposing a task-aware instruction network (TIN-SLT) with a learning-based feature fusion strategy and a multi-level data augmentation scheme, achieving BLEU-4 score improvements of 1.65 and 1.42 on PHOENIX-2014-T and ASLG-PC12 datasets.
Sign language recognition and translation first uses a recognition module to generate glosses from sign language videos and then employs a translation module to translate glosses into spoken sentences. Most existing works focus on the recognition step, while paying less attention to sign language translation. In this work, we propose a task-aware instruction network, namely TIN-SLT, for sign language translation, by introducing the instruction module and the learning-based feature fuse strategy into a Transformer network. In this way, the pre-trained model's language ability can be well explored and utilized to further boost the translation performance. Moreover, by exploring the representation space of sign language glosses and target spoken language, we propose a multi-level data augmentation scheme to adjust the data distribution of the training set. We conduct extensive experiments on two challenging benchmark datasets, PHOENIX-2014-T and ASLG-PC12, on which our method outperforms former best solutions by 1.65 and 1.42 in terms of BLEU-4. Our code is published at https://github.com/yongcaoplus/TIN-SLT.