Speed Co-Augmentation for Unsupervised Audio-Visual Pre-training
Jiangliu Wang Jianbo Jiao Yibing Song Stephen James Zhan Tong Chongjian Ge Pieter Abbeel Yun-hui Liu
This work aims to improve unsupervised audio-visual pre-training. Inspired by
the efficacy of data augmentation in visual contrastive learning, we propose a
novel speed co-augmentation method that randomly changes the playback speeds of
both audio and video data. Despite its simplicity, the speed co-augmentation
method possesses two compelling attributes: (1) it increases the diversity of
audio-visual pairs and doubles the size of negative pairs, resulting in a
significant enhancement in the learned representations, and (2) it changes the
strict correlation between audio-visual pairs but introduces a partial
relationship between the augmented pairs, which is modeled by our proposed
SoftInfoNCE loss to further boost the performance. Experimental results show
that the proposed method significantly improves the learned representations
when compared to vanilla audio-visual contrastive learning.