MAVD: The First Open Large-Scale Mandarin Audio-Visual Dataset with
  Depth Information
Jianrong Wang Yuchen Huo Li Liu Tianyi Xu Qi Li Sen Li
Audio-visual speech recognition (AVSR) gains increasing attention from
researchers as an important part of human-computer interaction. However, the
existing available Mandarin audio-visual datasets are limited and lack the
depth information. To address this issue, this work establishes the MAVD, a new
large-scale Mandarin multimodal corpus comprising 12,484 utterances spoken by
64 native Chinese speakers. To ensure the dataset covers diverse real-world
scenarios, a pipeline for cleaning and filtering the raw text material has been
developed to create a well-balanced reading material. In particular, the latest
data acquisition device of Microsoft, Azure Kinect is used to capture depth
information in addition to the traditional audio signals and RGB images during
data acquisition. We also provide a baseline experiment, which could be used to
evaluate the effectiveness of the dataset. The dataset and code will be
released at https://github.com/SpringHuo/MAVD.