Geo-Sign: Hyperbolic Contrastive Regularisation for Geometrically Aware Sign Language Translation
This work addresses the challenge of fine-grained motion representation in Sign Language Translation, offering a novel geometric approach that could benefit accessibility technologies, though it appears incremental by building on existing skeletal and hyperbolic methods.
The authors tackled the problem of improving geometric properties in skeletal representations for Sign Language Translation by proposing Geo-Sign, which uses hyperbolic geometry to model hierarchical sign language kinematics, resulting in enhanced embeddings that outperform state-of-the-art RGB methods while preserving privacy and computational efficiency.
Recent progress in Sign Language Translation (SLT) has focussed primarily on improving the representational capacity of large language models to incorporate Sign Language features. This work explores an alternative direction: enhancing the geometric properties of skeletal representations themselves. We propose Geo-Sign, a method that leverages the properties of hyperbolic geometry to model the hierarchical structure inherent in sign language kinematics. By projecting skeletal features derived from Spatio-Temporal Graph Convolutional Networks (ST-GCNs) into the Poincaré ball model, we aim to create more discriminative embeddings, particularly for fine-grained motions like finger articulations. We introduce a hyperbolic projection layer, a weighted Fréchet mean aggregation scheme, and a geometric contrastive loss operating directly in hyperbolic space. These components are integrated into an end-to-end translation framework as a regularisation function, to enhance the representations within the language model. This work demonstrates the potential of hyperbolic geometry to improve skeletal representations for Sign Language Translation, improving on SOTA RGB methods while preserving privacy and improving computational efficiency. Code available here: https://github.com/ed-fish/geo-sign.