Selective Contrastive Learning For Gloss Free Sign Language Translation
For researchers in sign language translation, this work provides a method to improve cross-modal alignment by selecting informative negatives, though it is an incremental improvement over existing contrastive learning approaches.
The paper addresses the problem of noisy and uninformative negative pairs in contrastive learning for gloss-free sign language translation. The proposed Selective Contrastive Learning (SCL-SLT) with a Pair Selection strategy improves alignment, achieving state-of-the-art results on two benchmarks (e.g., +1.5 BLEU on PHOENIX-2014T).
Sign language translation (SLT) converts continuous sign videos into spoken-language text, yet it remains challenging due to the intrinsic modality mismatch between visual signs and written text, particularly in gloss-free settings. Recent SLT systems increasingly adopt CLIP-like Vision-Language pretraining (VLP) for cross-modal alignment, but the random in-batch contrast provides few, batch-dependent negatives and may mislabel semantically similar (or even identical) pairs as negatives, introducing noisy and potentially inconsistent alignment supervision. In this work, we first conduct a preliminary trajectory-based analysis that tracks negative video-text similarity over training. The results show that only a small subset of negatives exhibits the desired behavior of being consistently pushed away, while the remaining negatives display heterogeneous and often non-decreasing similarity dynamics, suggesting that random in-batch negatives are frequently uninformative for effective alignment. Inspired by this, we propose Selective Contrastive Learning for SLT (SCL-SLT) with a Pair Selection (PS) strategy. PS scores candidate negatives using similarity dynamics from reference checkpoints and constructs mini-batches via a curriculum that progressively emphasizes more challenging negatives, thereby strengthening contrastive supervision while reducing the influence of noisy or semantically invalid negatives.