Reduce, Reuse, Recycle: Is Perturbed Data better than Other Language
  augmentation for Low Resource Self-Supervised Speech Models
Asad Ullah Alessandro Ragano Andrew Hines
Self-supervised representation learning (SSRL) has improved the performance
on downstream phoneme recognition versus supervised models. Training SSRL
models requires a large amount of pre-training data and this poses a challenge
for low resource languages. A common approach is transferring knowledge from
other languages. Instead, we propose to use audio augmentation to pre-train
SSRL models in a low resource condition and evaluate phoneme recognition as
downstream task. We performed a systematic comparison of augmentation
techniques, namely: pitch variation, noise addition, accented target-language
speech and other language speech. We found combined augmentations (noise/pitch)
was the best augmentation strategy outperforming accent and language knowledge
transfer. We compared the performance with various quantities and types of
pre-training data. We examined the scaling factor of augmented data to achieve
equivalent performance to models pre-trained with target domain speech. Our
findings suggest that for resource constrained languages, in-domain synthetic
augmentation can outperform knowledge transfer from accented or other language
speech.