Improvement in Sign Language Translation Using Text CTC Alignment
This work improves sign language translation for accessibility applications, but it is incremental as it builds on existing joint CTC/Attention and transfer learning methods.
The paper tackles the problem of sign language translation by addressing the limitation of gloss-based supervision with CTC in handling non-monotonic alignments between sign language video and spoken text, achieving results comparable to state-of-the-art on benchmarks like RWTH-PHOENIX-Weather 2014 T and CSL-Daily.
Current sign language translation (SLT) approaches often rely on gloss-based supervision with Connectionist Temporal Classification (CTC), limiting their ability to handle non-monotonic alignments between sign language video and spoken text. In this work, we propose a novel method combining joint CTC/Attention and transfer learning. The joint CTC/Attention introduces hierarchical encoding and integrates CTC with the attention mechanism during decoding, effectively managing both monotonic and non-monotonic alignments. Meanwhile, transfer learning helps bridge the modality gap between vision and language in SLT. Experimental results on two widely adopted benchmarks, RWTH-PHOENIX-Weather 2014 T and CSL-Daily, show that our method achieves results comparable to state-of-the-art and outperforms the pure-attention baseline. Additionally, this work opens a new door for future research into gloss-free SLT using text-based CTC alignment.