Reducing Barriers to Self-Supervised Learning: HuBERT Pre-training with
  Academic Compute
William Chen Xuankai Chang Yifan Peng Zhaoheng Ni Soumi Maiti Shinji Watanabe
Self-supervised learning (SSL) has led to great strides in speech processing.
However, the resources needed to train these models has become prohibitively
large as they continue to scale. Currently, only a few groups with substantial
resources are capable of creating SSL models, which harms reproducibility. In
this work, we optimize HuBERT SSL to fit in academic constraints. We reproduce
HuBERT independently from the original implementation, with no performance
loss. Our code and training optimizations make SSL feasible with only 8 GPUs,
instead of the 32 used in the original work. We also explore a semi-supervised
route, using an ASR model to skip the first pre-training iteration. Within one
iteration of pre-training, our models improve over HuBERT on several tasks.
Furthermore, our HuBERT Large variant requires only 8 GPUs, achieving similar
performance to the original trained on 128. As our contribution to the
community, all models, configurations, and code are made open-source in ESPnet.