Text-to-feature diffusion for audio-visual few-shot learning
Otniel-Bogdan Mercea Thomas Hummel A. Sophia Koepke Zeynep Akata
Training deep learning models for video classification from audio-visual data
commonly requires immense amounts of labeled training data collected via a
costly process. A challenging and underexplored, yet much cheaper, setup is
few-shot learning from video data. In particular, the inherently multi-modal
nature of video data with sound and visual information has not been leveraged
extensively for the few-shot video classification task. Therefore, we introduce
a unified audio-visual few-shot video classification benchmark on three
datasets, i.e. the VGGSound-FSL, UCF-FSL, ActivityNet-FSL datasets, where we
adapt and compare ten methods. In addition, we propose AV-DIFF, a
text-to-feature diffusion framework, which first fuses the temporal and
audio-visual features via cross-modal attention and then generates multi-modal
features for the novel classes. We show that AV-DIFF obtains state-of-the-art
performance on our proposed benchmark for audio-visual (generalised) few-shot
learning. Our benchmark paves the way for effective audio-visual classification
when only limited labeled data is available. Code and data are available at
https://github.com/ExplainableML/AVDIFF-GFSL.