SAGE: Segment-Aware Gloss-Free Encoding for Token-Efficient Sign Language Translation
This addresses scalability issues for sign language translation systems, especially as datasets grow, by reducing computational costs without gloss annotations, though it is incremental in improving existing methods.
The paper tackled the problem of high computational demands in gloss-free sign language translation by proposing a segment-aware visual tokenization framework, which reduced input sequence length by up to 50% and memory usage by up to 2.67x while exceeding state-of-the-art performance on the PHOENIX14T benchmark.
Gloss-free Sign Language Translation (SLT) has advanced rapidly, achieving strong performances without relying on gloss annotations. However, these gains have often come with increased model complexity and high computational demands, raising concerns about scalability, especially as large-scale sign language datasets become more common. We propose a segment-aware visual tokenization framework that leverages sign segmentation to convert continuous video into discrete, sign-informed visual tokens. This reduces input sequence length by up to 50% compared to prior methods, resulting in up to 2.67x lower memory usage and better scalability on larger datasets. To bridge the visual and linguistic modalities, we introduce a token-to-token contrastive alignment objective, along with a dual-level supervision that aligns both language embeddings and intermediate hidden states. This improves fine-grained cross-modal alignment without relying on gloss-level supervision. Our approach notably exceeds the performance of state-of-the-art methods on the PHOENIX14T benchmark, while significantly reducing sequence length. Further experiments also demonstrate our improved performance over prior work under comparable sequence-lengths, validating the potential of our tokenization and alignment strategies.