Ham2Pose: Animating Sign Language Notation into Pose Sequences
This work addresses the need for universal sign language animation to facilitate communication between hearing and hearing-impaired communities, though it is incremental in improving pose sequence generation and evaluation.
The paper tackles the problem of translating HamNoSys notation into sign language pose sequences, proposing a transformer-based method that achieves this with weak supervision and introduces a new distance measurement, nDTW, validated on a large-scale dataset.
Translating spoken languages into Sign languages is necessary for open communication between the hearing and hearing-impaired communities. To achieve this goal, we propose the first method for animating a text written in HamNoSys, a lexical Sign language notation, into signed pose sequences. As HamNoSys is universal, our proposed method offers a generic solution invariant to the target Sign language. Our method gradually generates pose predictions using transformer encoders that create meaningful representations of the text and poses while considering their spatial and temporal information. We use weak supervision for the training process and show that our method succeeds in learning from partial and inaccurate data. Additionally, we offer a new distance measurement for pose sequences, normalized Dynamic Time Warping (nDTW), based on DTW over normalized keypoints trajectories, and validate its correctness using AUTSL, a large-scale Sign language dataset. We show that it measures the distance between pose sequences more accurately than existing measurements and use it to assess the quality of our generated pose sequences. Code for the data pre-processing, the model, and the distance measurement is publicly released for future research.