The VVAD-LRS3 Dataset for Visual Voice Activity Detection
This addresses the problem of enabling more natural human-machine interaction for robotics by providing a large-scale dataset for VVAD, though it is incremental as it builds on existing data.
The authors tackled the lack of labeled data for Visual Voice Activity Detection (VVAD) by creating the VVAD-LRS3 dataset, which contains over 44K samples and is over three times larger than the next competitive dataset, achieving 92% accuracy with a CNN LSTM model on facial images.
Robots are becoming everyday devices, increasing their interaction with humans. To make human-machine interaction more natural, cognitive features like Visual Voice Activity Detection (VVAD), which can detect whether a person is speaking or not, given visual input of a camera, need to be implemented. Neural networks are state of the art for tasks in Image Processing, Time Series Prediction, Natural Language Processing and other domains. Those Networks require large quantities of labeled data. Currently there are not many datasets for the task of VVAD. In this work we created a large scale dataset called the VVAD-LRS3 dataset, derived by automatic annotations from the LRS3 dataset. The VVAD-LRS3 dataset contains over 44K samples, over three times the next competitive dataset (WildVVAD). We evaluate different baselines on four kinds of features: facial and lip images, and facial and lip landmark features. With a Convolutional Neural Network Long Short Term Memory (CNN LSTM) on facial images an accuracy of 92% was reached on the test set. A study with humans showed that they reach an accuracy of 87.93% on the test set.