3D Lip Event Detection via Interframe Motion Divergence at Multiple Temporal Resolutions
This work addresses lip event detection for speech analysis and support for the hearing impaired, representing an incremental improvement with a novel method for a known bottleneck.
The paper tackled the problem of automatically detecting lip events from 3D speaking lip sequences by proposing a pipeline that uses motion divergence and multi-temporal-resolution frameworks, achieving state-of-the-art performance in detecting opening and closing events across 100 sequences.
The lip is a dominant dynamic facial unit when a person is speaking. Detecting lip events is beneficial to speech analysis and support for the hearing impaired. This paper proposes a 3D lip event detection pipeline that automatically determines the lip events from a 3D speaking lip sequence. We define a motion divergence measure using 3D lip landmarks to quantify the interframe dynamics of a 3D speaking lip. Then, we cast the interframe motion detection in a multi-temporal-resolution framework that allows the detection to be applicable to different speaking speeds. The experiments on the S3DFM Dataset investigate the overall 3D lip dynamics based on the proposed motion divergence. The proposed 3D pipeline is able to detect opening and closing lip events across 100 sequences, achieving a state-of-the-art performance.