Towards Improving the Performance of Pre-Trained Speech Models for
  Low-Resource Languages Through Lateral Inhibition
Andrei-Marius Avram Răzvan-Alexandru Smădu Vasile Păiş Dumitru-Clementin Cercel Radu Ion Dan Tufiş
With the rise of bidirectional encoder representations from Transformer
models in natural language processing, the speech community has adopted some of
their development methodologies. Therefore, the Wav2Vec models were introduced
to reduce the data required to obtain state-of-the-art results. This work
leverages this knowledge and improves the performance of the pre-trained speech
models by simply replacing the fine-tuning dense layer with a lateral
inhibition layer inspired by the biological process. Our experiments on
Romanian, a low-resource language, show an average improvement of 12.5% word
error rate (WER) using the lateral inhibition layer. In addition, we obtain
state-of-the-art results on both the Romanian Speech Corpus and the Robin
Technical Acquisition Corpus with 1.78% WER and 29.64% WER, respectively.