Weakly-supervised Fingerspelling Recognition in British Sign Language Videos
This work addresses the lack of methods for BSL fingerspelling recognition, which is important for sign language research and accessibility, but it is incremental as it adapts existing techniques to a new domain with weak supervision.
The paper tackles the problem of detecting and recognizing fingerspelling in British Sign Language videos using only weak annotations from subtitles for training, achieving performance verified through extensive evaluations and providing a new expert-annotated test set of 5K clips.
The goal of this work is to detect and recognize sequences of letters signed using fingerspelling in British Sign Language (BSL). Previous fingerspelling recognition methods have not focused on BSL, which has a very different signing alphabet (e.g., two-handed instead of one-handed) to American Sign Language (ASL). They also use manual annotations for training. In contrast to previous methods, our method only uses weak annotations from subtitles for training. We localize potential instances of fingerspelling using a simple feature similarity method, then automatically annotate these instances by querying subtitle words and searching for corresponding mouthing cues from the signer. We propose a Transformer architecture adapted to this task, with a multiple-hypothesis CTC loss function to learn from alternative annotation possibilities. We employ a multi-stage training approach, where we make use of an initial version of our trained model to extend and enhance our training data before re-training again to achieve better performance. Through extensive evaluations, we verify our method for automatic annotation and our model architecture. Moreover, we provide a human expert annotated test set of 5K video clips for evaluating BSL fingerspelling recognition methods to support sign language research.