Integrating Facial Generation into Full-Duplex Spoken Dialogue Systems
This work addresses the lack of facial expression generation in full-duplex spoken dialogue systems, enhancing human-computer interaction for users of such systems.
Moshi-Face extends the full-duplex spoken dialogue model Moshi to jointly process user audio and facial input while generating synchronized speech and facial motion in real time, achieving audiovisual alignment at low latency without degrading dialogue quality.
Full-duplex spoken dialogue models, such as Moshi, enable natural, low-latency voice conversations. However, they remain limited to the audio modality, lacking the facial expressions that are integral to human communication. We present Moshi-Face, the first full-duplex dialogue model that jointly processes the user's audio and facial input while simultaneously generating speech and facial motion. We first construct a vector-quantized variational autoencoder (VQ-VAE) as a face codec that encodes 3D head meshes extracted from facial videos into compact discrete tokens, referred to as face tokens, and conversely reconstructs 3D meshes from these tokens. We then extend Moshi with a Face Transformer module that generates face tokens non-autoregressively, enabling Moshi-Face to produce synchronized audio and face tokens in real time. Experiments show that Moshi-Face achieves audiovisual alignment at low latency while preserving the dialogue quality of the original audio-only model.