When Less Is More: A Sparse Facial Motion Structure For Listening Motion Learning
This work addresses the challenge of inefficient and low-fidelity facial motion modeling in dyadic conversations, which is incremental as it builds on existing discrete representation approaches.
The paper tackles the problem of predicting non-verbal facial motion for listening head behavior in human-robot interaction by proposing a sparse representation using keyframes and transition frames, resulting in improved motion fidelity and diversity compared to state-of-the-art discrete token methods.
Effective human behavior modeling is critical for successful human-robot interaction. Current state-of-the-art approaches for predicting listening head behavior during dyadic conversations employ continuous-to-discrete representations, where continuous facial motion sequence is converted into discrete latent tokens. However, non-verbal facial motion presents unique challenges owing to its temporal variance and multi-modal nature. State-of-the-art discrete motion token representation struggles to capture underlying non-verbal facial patterns making training the listening head inefficient with low-fidelity generated motion. This study proposes a novel method for representing and predicting non-verbal facial motion by encoding long sequences into a sparse sequence of keyframes and transition frames. By identifying crucial motion steps and interpolating intermediate frames, our method preserves the temporal structure of motion while enhancing instance-wise diversity during the learning process. Additionally, we apply this novel sparse representation to the task of listening head prediction, demonstrating its contribution to improving the explanation of facial motion patterns.