Stable Distillation: Regularizing Continued Pre-training for
  Low-Resource Automatic Speech Recognition
Ashish Seth Sreyan Ghosh S. Umesh Dinesh Manocha
Continued self-supervised (SSL) pre-training for adapting existing SSL models
to the target domain has shown to be extremely effective for low-resource
Automatic Speech Recognition (ASR). This paper proposes Stable Distillation, a
simple and novel approach for SSL-based continued pre-training that boosts ASR
performance in the target domain where both labeled and unlabeled data are
limited. Stable Distillation employs self-distillation as regularization for
continued pre-training, alleviating the over-fitting issue, a common problem
continued pre-training faces when the source and target domains differ.
Specifically, first, we perform vanilla continued pre-training on an initial
SSL pre-trained model on the target domain ASR dataset and call it the teacher.
Next, we take the same initial pre-trained model as a student to perform
continued pre-training while enforcing its hidden representations to be close
to that of the teacher (via MSE loss). This student is then used for downstream
ASR fine-tuning on the target dataset. In practice, Stable Distillation
outperforms all our baselines by 0.8 - 7 WER when evaluated in various
experimental settings.