M3T: Discrete Multi-Modal Motion Tokens for Sign Language Production
This addresses the challenge of producing grammatically complete sign language for deaf and hard-of-hearing communities, representing a strong specific gain rather than an incremental improvement.
The paper tackled the problem of generating sign language with non-manual features by proposing M3T, a system that uses SMPL-FX and modality-specific tokenization, achieving state-of-the-art quality on benchmarks and 58.3% accuracy on a dataset where signs rely on non-manual features.
Sign language production requires more than hand motion generation. Non-manual features, including mouthings, eyebrow raises, gaze, and head movements, are grammatically obligatory and cannot be recovered from manual articulators alone. Existing 3D production systems face two barriers to integrating them: the standard body model provides a facial space too low-dimensional to encode these articulations, and when richer representations are adopted, standard discrete tokenization suffers from codebook collapse, leaving most of the expression space unreachable. We propose SMPL-FX, which couples FLAME's rich expression space with the SMPL-X body, and tokenize the resulting representation with modality-specific Finite Scalar Quantization VAEs for body, hands, and face. M3T is an autoregressive transformer trained on this multi-modal motion vocabulary, with an auxiliary translation objective that encourages semantically grounded embeddings. Across three standard benchmarks (How2Sign, CSL-Daily, Phoenix14T) M3T achieves state-of-the-art sign language production quality, and on NMFs-CSL, where signs are distinguishable only by non-manual features, reaches 58.3% accuracy against 49.0% for the strongest comparable pose baseline.