Looking for the Signs: Identifying Isolated Sign Instances in Continuous Video Footage
This work addresses the challenge of identifying isolated signs in continuous video footage for sign language recognition applications, representing a strong specific gain in this domain.
The paper tackles the problem of one-shot sign spotting in continuous sign language videos by proposing SignLookup, a transformer-based network that uses attention mechanisms to handle temporal scale discrepancies. The model achieves state-of-the-art performance with up to 96% accuracy on benchmark datasets and generalizes across different sign languages.
In this paper, we focus on the task of one-shot sign spotting, i.e. given an example of an isolated sign (query), we want to identify whether/where this sign appears in a continuous, co-articulated sign language video (target). To achieve this goal, we propose a transformer-based network, called SignLookup. We employ 3D Convolutional Neural Networks (CNNs) to extract spatio-temporal representations from video clips. To solve the temporal scale discrepancies between the query and the target videos, we construct multiple queries from a single video clip using different frame-level strides. Self-attention is applied across these query clips to simulate a continuous scale space. We also utilize another self-attention module on the target video to learn the contextual within the sequence. Finally a mutual-attention is used to match the temporal scales to localize the query within the target sequence. Extensive experiments demonstrate that the proposed approach can not only reliably identify isolated signs in continuous videos, regardless of the signers' appearance, but can also generalize to different sign languages. By taking advantage of the attention mechanism and the adaptive features, our model achieves state-of-the-art performance on the sign spotting task with accuracy as high as 96% on challenging benchmark datasets and significantly outperforming other approaches.