Gloss2Text: Sign Language Gloss translation using LLMs and Semantically Aware Label Smoothing
This addresses sign language translation for improved accessibility, though it is incremental as it focuses on one stage of a multi-stage pipeline.
The paper tackles sign language translation from gloss annotations to spoken text by leveraging pre-trained LLMs, data augmentation, and a novel label-smoothing loss function, achieving state-of-the-art performance on the PHOENIX Weather 2014T dataset.
Sign language translation from video to spoken text presents unique challenges owing to the distinct grammar, expression nuances, and high variation of visual appearance across different speakers and contexts. The intermediate gloss annotations of videos aim to guide the translation process. In our work, we focus on {\em Gloss2Text} translation stage and propose several advances by leveraging pre-trained large language models (LLMs), data augmentation, and novel label-smoothing loss function exploiting gloss translation ambiguities improving significantly the performance of state-of-the-art approaches. Through extensive experiments and ablation studies on the PHOENIX Weather 2014T dataset, our approach surpasses state-of-the-art performance in {\em Gloss2Text} translation, indicating its efficacy in addressing sign language translation and suggesting promising avenues for future research and development.