An Efficient Sign Language Translation Using Spatial Configuration and Motion Dynamics with LLMs
This work addresses sign language translation for accessibility applications, presenting an incremental improvement by focusing on spatial and motion features without domain-specific fine-tuning.
The paper tackled the problem of gloss-free sign language translation by introducing SpaMo, a framework that captures spatial configurations and motion dynamics using off-the-shelf visual encoders and LLMs, achieving state-of-the-art performance on PHOENIX14T and How2Sign datasets.
Gloss-free Sign Language Translation (SLT) converts sign videos directly into spoken language sentences without relying on glosses. Recently, Large Language Models (LLMs) have shown remarkable translation performance in gloss-free methods by harnessing their powerful natural language generation capabilities. However, these methods often rely on domain-specific fine-tuning of visual encoders to achieve optimal results. By contrast, this paper emphasizes the importance of capturing the spatial configurations and motion dynamics inherent in sign language. With this in mind, we introduce Spatial and Motion-based Sign Language Translation (SpaMo), a novel LLM-based SLT framework. The core idea of SpaMo is simple yet effective. We first extract spatial and motion features using off-the-shelf visual encoders and then input these features into an LLM with a language prompt. Additionally, we employ a visual-text alignment process as a warm-up before the SLT supervision. Our experiments demonstrate that SpaMo achieves state-of-the-art performance on two popular datasets, PHOENIX14T and How2Sign.