Modeling Intensification for Sign Language Generation: A Computational Approach
This work addresses the challenge of generating more natural and understandable sign language for deaf and hard-of-hearing communities, though it is incremental as it builds on existing transformer models with enhanced data.
The paper tackled the problem of poor prosody in end-to-end sign language generation by modeling intensification through data-driven strategies and enhanced annotations on the PHOENIX-14T dataset, resulting in improved automatic metrics and higher human preference for generated videos.
End-to-end sign language generation models do not accurately represent the prosody in sign language. A lack of temporal and spatial variations leads to poor-quality generated presentations that confuse human interpreters. In this paper, we aim to improve the prosody in generated sign languages by modeling intensification in a data-driven manner. We present different strategies grounded in linguistics of sign language that inform how intensity modifiers can be represented in gloss annotations. To employ our strategies, we first annotate a subset of the benchmark PHOENIX-14T, a German Sign Language dataset, with different levels of intensification. We then use a supervised intensity tagger to extend the annotated dataset and obtain labels for the remaining portion of it. This enhanced dataset is then used to train state-of-the-art transformer models for sign language generation. We find that our efforts in intensification modeling yield better results when evaluated with automatic metrics. Human evaluation also indicates a higher preference of the videos generated using our model.