Diff-TTSG: Denoising probabilistic integrated speech and gesture synthesis
This work addresses the challenge of generating realistic spontaneous human communication for applications like virtual agents or animation, representing a novel but incremental advance in multimodal synthesis.
The authors tackled the problem of jointly synthesizing spontaneous speech and co-speech gestures, which previous non-probabilistic methods struggled with due to variability and artifacts, and introduced Diff-TTSG, a diffusion-based probabilistic model that achieved improved synthesis quality as validated through subjective tests.
With read-aloud speech synthesis achieving high naturalness scores, there is a growing research interest in synthesising spontaneous speech. However, human spontaneous face-to-face conversation has both spoken and non-verbal aspects (here, co-speech gestures). Only recently has research begun to explore the benefits of jointly synthesising these two modalities in a single system. The previous state of the art used non-probabilistic methods, which fail to capture the variability of human speech and motion, and risk producing oversmoothing artefacts and sub-optimal synthesis quality. We present the first diffusion-based probabilistic model, called Diff-TTSG, that jointly learns to synthesise speech and gestures together. Our method can be trained on small datasets from scratch. Furthermore, we describe a set of careful uni- and multi-modal subjective tests for evaluating integrated speech and gesture synthesis systems, and use them to validate our proposed approach. Please see https://shivammehta25.github.io/Diff-TTSG/ for video examples, data, and code.