Diversity-Aware Sign Language Production through a Pose Encoding Variational Autoencoder
This work addresses the need for more inclusive and varied sign language production, though it appears incremental as it builds on existing variational inference and UNet architectures.
The paper tackles the problem of generating diverse sign language images by modifying attributes like gender and skin color while preserving pose, using a pose encoding variational autoencoder to improve synthesis quality. Experiments on the SMILE II dataset show quantitative improvements over state-of-the-art baselines in diversity, image quality, and pose estimation.
This paper addresses the problem of diversity-aware sign language production, where we want to give an image (or sequence) of a signer and produce another image with the same pose but different attributes (\textit{e.g.} gender, skin color). To this end, we extend the variational inference paradigm to include information about the pose and the conditioning of the attributes. This formulation improves the quality of the synthesised images. The generator framework is presented as a UNet architecture to ensure spatial preservation of the input pose, and we include the visual features from the variational inference to maintain control over appearance and style. We generate each body part with a separate decoder. This architecture allows the generator to deliver better overall results. Experiments on the SMILE II dataset show that the proposed model performs quantitatively better than state-of-the-art baselines regarding diversity, per-pixel image quality, and pose estimation. Quantitatively, it faithfully reproduces non-manual features for signers.