Real-Time Lip Sync for Live 2D Animation
This addresses the need for fast and accurate lip sync in live broadcasts and streaming platforms, enabling 2D characters to respond naturally to human performers, though it is incremental as it builds on existing LSTM and data augmentation techniques.
The paper tackles real-time lip sync for live 2D animation by developing a deep learning system using an LSTM model that processes streaming audio with less than 200ms latency, achieving results preferred over competing methods in human judgement experiments.
The emergence of commercial tools for real-time performance-based 2D animation has enabled 2D characters to appear on live broadcasts and streaming platforms. A key requirement for live animation is fast and accurate lip sync that allows characters to respond naturally to other actors or the audience through the voice of a human performer. In this work, we present a deep learning based interactive system that automatically generates live lip sync for layered 2D characters using a Long Short Term Memory (LSTM) model. Our system takes streaming audio as input and produces viseme sequences with less than 200ms of latency (including processing time). Our contributions include specific design decisions for our feature definition and LSTM configuration that provide a small but useful amount of lookahead to produce accurate lip sync. We also describe a data augmentation procedure that allows us to achieve good results with a very small amount of hand-animated training data (13-20 minutes). Extensive human judgement experiments show that our results are preferred over several competing methods, including those that only support offline (non-live) processing. Video summary and supplementary results at GitHub link: https://github.com/deepalianeja/CharacterLipSync2D