Multitask training with unlabeled data for end-to-end sign language fingerspelling recognition
This addresses the problem of data scarcity and reliance on manual annotations for sign language recognition, offering a more efficient approach for accessibility applications.
The paper tackles automatic American Sign Language fingerspelling recognition from video by introducing an end-to-end model that avoids frame-level labels and hand-crafted features, achieving 11.6% and 4.4% absolute letter accuracy improvements in signer-independent and signer-adapted settings over prior methods.
We address the problem of automatic American Sign Language fingerspelling recognition from video. Prior work has largely relied on frame-level labels, hand-crafted features, or other constraints, and has been hampered by the scarcity of data for this task. We introduce a model for fingerspelling recognition that addresses these issues. The model consists of an auto-encoder-based feature extractor and an attention-based neural encoder-decoder, which are trained jointly. The model receives a sequence of image frames and outputs the fingerspelled word, without relying on any frame-level training labels or hand-crafted features. In addition, the auto-encoder subcomponent makes it possible to leverage unlabeled data to improve the feature learning. The model achieves 11.6% and 4.4% absolute letter accuracy improvement respectively in signer-independent and signer-adapted fingerspelling recognition over previous approaches that required frame-level training labels.