Stable Signer: Hierarchical Sign Language Generative Model
This addresses the slow progress in generating accurate and high-quality sign language videos, which is crucial for accessibility and communication for the deaf and hard-of-hearing community, though it appears incremental as it builds on existing stages.
The paper tackles the problem of Sign Language Production (SLP) by streamlining the traditional multi-stage pipeline into a hierarchical end-to-end model called Stable Signer, which improves performance by 48.6% compared to current state-of-the-art methods.
Sign Language Production (SLP) is the process of converting the complex input text into a real video. Most previous works focused on the Text2Gloss, Gloss2Pose, Pose2Vid stages, and some concentrated on Prompt2Gloss and Text2Avatar stages. However, this field has made slow progress due to the inaccuracy of text conversion, pose generation, and the rendering of poses into real human videos in these stages, resulting in gradually accumulating errors. Therefore, in this paper, we streamline the traditional redundant structure, simplify and optimize the task objective, and design a new sign language generative model called Stable Signer. It redefines the SLP task as a hierarchical generation end-to-end task that only includes text understanding (Prompt2Gloss, Text2Gloss) and Pose2Vid, and executes text understanding through our proposed new Sign Language Understanding Linker called SLUL, and generates hand gestures through the named SLP-MoE hand gesture rendering expert block to end-to-end generate high-quality and multi-style sign language videos. SLUL is trained using the newly developed Semantic-Aware Gloss Masking Loss (SAGM Loss). Its performance has improved by 48.6% compared to the current SOTA generation methods.