DyStream: Streaming Dyadic Talking Heads Generation via Flow Matching-based Autoregressive Model
This addresses the need for immediate, non-verbal feedback in interactive applications by reducing latency, though it is incremental as it builds on existing methods with novel enhancements.
The paper tackled the problem of generating realistic dyadic talking head videos with ultra-low latency, achieving a generation time of 34 ms per frame and state-of-the-art lip-sync quality scores of 8.13 and 7.61 on HDTF.
Generating realistic, dyadic talking head video requires ultra-low latency. Existing chunk-based methods require full non-causal context windows, introducing significant delays. This high latency critically prevents the immediate, non-verbal feedback required for a realistic listener. To address this, we present DyStream, a flow matching-based autoregressive model that could generate video in real-time from both speaker and listener audio. Our method contains two key designs: (1) we adopt a stream-friendly autoregressive framework with flow-matching heads for probabilistic modeling, and (2) We propose a causal encoder enhanced by a lookahead module to incorporate short future context (e.g., 60 ms) to improve quality while maintaining low latency. Our analysis shows this simple-and-effective method significantly surpass alternative causal strategies, including distillation and generative encoder. Extensive experiments show that DyStream could generate video within 34 ms per frame, guaranteeing the entire system latency remains under 100 ms. Besides, it achieves state-of-the-art lip-sync quality, with offline and online LipSync Confidence scores of 8.13 and 7.61 on HDTF, respectively. The model, weights and codes are available.