Deep Understanding of Sign Language for Sign to Subtitle Alignment
This work addresses the problem of sign language video alignment and translation for accessibility, particularly in scenarios where manual labeling is challenging, though it appears incremental as it builds on existing methods with specific enhancements.
The paper tackles aligning asynchronous subtitles in sign language videos with limited labeled data by leveraging BSL grammatical rules, a selective alignment loss, and self-training with refined pseudo-labels, achieving state-of-the-art results with substantial improvements in frame-level accuracy and F1-score.
The objective of this work is to align asynchronous subtitles in sign language videos with limited labelled data. To achieve this goal, we propose a novel framework with the following contributions: (1) we leverage fundamental grammatical rules of British Sign Language (BSL) to pre-process the input subtitles, (2) we design a selective alignment loss to optimise the model for predicting the temporal location of signs only when the queried sign actually occurs in a scene, and (3) we conduct self-training with refined pseudo-labels which are more accurate than the heuristic audio-aligned labels. From this, our model not only better understands the correlation between the text and the signs, but also holds potential for application in the translation of sign languages, particularly in scenarios where manual labelling of large-scale sign data is impractical or challenging. Extensive experimental results demonstrate that our approach achieves state-of-the-art results, surpassing previous baselines by substantial margins in terms of both frame-level accuracy and F1-score. This highlights the effectiveness and practicality of our framework in advancing the field of sign language video alignment and translation.