Meaningful Pose-Based Sign Language Evaluation
This provides a practical toolkit for developing and evaluating sign language translation systems, but it is incremental as it builds on existing evaluation methods.
The study tackled the problem of evaluating sign language utterances using skeletal poses by comparing keypoint distance, embedding, and back-translation metrics, showing tradeoffs in different scenarios through automatic and human studies.
We present a comprehensive study on meaningfully evaluating sign language utterances in the form of human skeletal poses. The study covers keypoint distance-based, embedding-based, and back-translation-based metrics. We show tradeoffs between different metrics in different scenarios through automatic meta-evaluation of sign-level retrieval and a human correlation study of text-to-pose translation across different sign languages. Our findings and the open-source pose-evaluation toolkit provide a practical and reproducible way of developing and evaluating sign language translation or generation systems.