CSF: Contrastive Semantic Features for Direct Multilingual Sign Language Generation
This addresses accessibility barriers for non-English speakers in the global deaf community by enabling real-time, direct sign language generation.
The paper tackles the problem of sign language translation systems requiring English as an intermediary by introducing Canonical Semantic Form (CSF), a language-agnostic semantic representation framework that enables direct translation from any source language to sign language, achieving 99.03% average slot extraction accuracy across four languages with 3.02ms inference latency on CPU.
Sign language translation systems typically require English as an intermediary language, creating barriers for non-English speakers in the global deaf community. We present Canonical Semantic Form (CSF), a language-agnostic semantic representation framework that enables direct translation from any source language to sign language without English mediation. CSF decomposes utterances into nine universal semantic slots: event, intent, time, condition, agent, object, location, purpose, and modifier. A key contribution is our comprehensive condition taxonomy comprising 35 condition types across eight semantic categories, enabling nuanced representation of conditional expressions common in everyday communication. We train a lightweight transformer-based extractor (0.74 MB) that achieves 99.03% average slot extraction accuracy across four typologically diverse languages: English, Vietnamese, Japanese, and French. The model demonstrates particularly strong performance on condition classification (99.4% accuracy) despite the 35-class complexity. With inference latency of 3.02ms on CPU, our approach enables real-time sign language generation in browser-based applications. We release our code, trained models, and multilingual dataset to support further research in accessible sign language technology.