Transfer Learning of Transformer-based Speech Recognition Models from
  Czech to Slovak
Jan Lehečka Josef V. Psutka Josef Psutka
In this paper, we are comparing several methods of training the Slovak speech
recognition models based on the Transformers architecture. Specifically, we are
exploring the approach of transfer learning from the existing Czech pre-trained
Wav2Vec 2.0 model into Slovak. We are demonstrating the benefits of the
proposed approach on three Slovak datasets. Our Slovak models scored the best
results when initializing the weights from the Czech model at the beginning of
the pre-training phase. Our results show that the knowledge stored in the Cezch
pre-trained model can be successfully reused to solve tasks in Slovak while
outperforming even much larger public multilingual models.