Audiovisual Moments in Time: A Large-Scale Annotated Dataset of
  Audiovisual Actions
Michael Joannou Pia Rotshtein Uta Noppeney
We present Audiovisual Moments in Time (AVMIT), a large-scale dataset of
audiovisual action events. In an extensive annotation task 11 participants
labelled a subset of 3-second audiovisual videos from the Moments in Time
dataset (MIT). For each trial, participants assessed whether the labelled
audiovisual action event was present and whether it was the most prominent
feature of the video. The dataset includes the annotation of 57,177 audiovisual
videos, each independently evaluated by 3 of 11 trained participants. From this
initial collection, we created a curated test set of 16 distinct action
classes, with 60 videos each (960 videos). We also offer 2 sets of pre-computed
audiovisual feature embeddings, using VGGish/YamNet for audio data and
VGG16/EfficientNetB0 for visual data, thereby lowering the barrier to entry for
audiovisual DNN research. We explored the advantages of AVMIT annotations and
feature embeddings to improve performance on audiovisual event recognition. A
series of 6 Recurrent Neural Networks (RNNs) were trained on either
AVMIT-filtered audiovisual events or modality-agnostic events from MIT, and
then tested on our audiovisual test set. In all RNNs, top 1 accuracy was
increased by 2.71-5.94\% by training exclusively on audiovisual events, even
outweighing a three-fold increase in training data. We anticipate that the
newly annotated AVMIT dataset will serve as a valuable resource for research
and comparative experiments involving computational models and human
participants, specifically when addressing research questions where audiovisual
correspondence is of critical importance.