Exploring Self-Supervised Contrastive Learning of Spatial Sound Event
  Representation
Xilin Jiang Cong Han Yinghao Aaron Li Nima Mesgarani
In this study, we present a simple multi-channel framework for contrastive
learning (MC-SimCLR) to encode 'what' and 'where' of spatial audios. MC-SimCLR
learns joint spectral and spatial representations from unlabeled spatial
audios, thereby enhancing both event classification and sound localization in
downstream tasks. At its core, we propose a multi-level data augmentation
pipeline that augments different levels of audio features, including waveforms,
Mel spectrograms, and generalized cross-correlation (GCC) features. In
addition, we introduce simple yet effective channel-wise augmentation methods
to randomly swap the order of the microphones and mask Mel and GCC channels. By
using these augmentations, we find that linear layers on top of the learned
representation significantly outperform supervised models in terms of both
event classification accuracy and localization error. We also perform a
comprehensive analysis of the effect of each augmentation method and a
comparison of the fine-tuning performance using different amounts of labeled
data.