Diverse Sign Language Translation
This addresses the need for more flexible and accurate sign language translation systems, particularly for users with limited data, though it is incremental as it builds on existing SLT methods.
The paper tackles the problem of rigid one-to-one mapping in sign language translation (SLT) by introducing a Diverse Sign Language Translation (DivSLT) task to generate multiple accurate translations, using LLMs to efficiently create enriched datasets and a benchmark model with multi-reference training and reinforcement learning, achieving improved translation performance and diversity on CSL-Daily and PHOENIX14T datasets.
Like spoken languages, a single sign language expression could correspond to multiple valid textual interpretations. Hence, learning a rigid one-to-one mapping for sign language translation (SLT) models might be inadequate, particularly in the case of limited data. In this work, we introduce a Diverse Sign Language Translation (DivSLT) task, aiming to generate diverse yet accurate translations for sign language videos. Firstly, we employ large language models (LLM) to generate multiple references for the widely-used CSL-Daily and PHOENIX14T SLT datasets. Here, native speakers are only invited to touch up inaccurate references, thus significantly improving the annotation efficiency. Secondly, we provide a benchmark model to spur research in this task. Specifically, we investigate multi-reference training strategies to enable our DivSLT model to achieve diverse translations. Then, to enhance translation accuracy, we employ the max-reward-driven reinforcement learning objective that maximizes the reward of the translated result. Additionally, we utilize multiple metrics to assess the accuracy, diversity, and semantic precision of the DivSLT task. Experimental results on the enriched datasets demonstrate that our DivSLT method achieves not only better translation performance but also diverse translation results.