MAVD: The First Open Large-Scale Mandarin Audio-Visual Dataset with Depth Information
This dataset addresses a gap for researchers in audio-visual speech recognition, particularly for Mandarin, but is incremental as it extends existing data collection with depth information.
The authors tackled the lack of a large-scale Mandarin audio-visual dataset with depth information by creating MAVD, which includes 12,484 utterances from 64 speakers and uses Azure Kinect for depth capture, providing a baseline for evaluation.
Audio-visual speech recognition (AVSR) gains increasing attention from researchers as an important part of human-computer interaction. However, the existing available Mandarin audio-visual datasets are limited and lack the depth information. To address this issue, this work establishes the MAVD, a new large-scale Mandarin multimodal corpus comprising 12,484 utterances spoken by 64 native Chinese speakers. To ensure the dataset covers diverse real-world scenarios, a pipeline for cleaning and filtering the raw text material has been developed to create a well-balanced reading material. In particular, the latest data acquisition device of Microsoft, Azure Kinect is used to capture depth information in addition to the traditional audio signals and RGB images during data acquisition. We also provide a baseline experiment, which could be used to evaluate the effectiveness of the dataset. The dataset and code will be released at https://github.com/SpringHuo/MAVD.