Teach Me Sign: Stepwise Prompting LLM for Sign Language Production
This addresses sign language production for accessibility applications, but it is incremental as it adapts existing LLM methods to a new domain.
The paper tackles sign language generation by fine-tuning a large language model (LLM) to treat sign language as a natural language, using stepwise prompting to extract inherent knowledge. Results on How2Sign and Phoenix14T datasets show the approach effectively aligns distributions and grammatical rules between sign and spoken language.
Large language models, with their strong reasoning ability and rich knowledge, have brought revolution to many tasks of AI, but their impact on sign language generation remains limited due to its complexity and unique rules. In this paper, we propose TEAch Me Sign (TEAM-Sign), treating sign language as another natural language. By fine-tuning an LLM, we enable it to learn the correspondence between text and sign language, and facilitate generation. Considering the differences between sign and spoken language, we employ a stepwise prompting strategy to extract the inherent sign language knowledge within the LLM, thereby supporting the learning and generation process. Experimental results on How2Sign and Phoenix14T datasets demonstrate that our approach effectively leverages both the sign language knowledge and reasoning capabilities of LLM to align the different distribution and grammatical rules between sign and spoken language.