Byte Pair Encoding Is All You Need For Automatic Bengali Speech
  Recognition
Ahnaf Mozib Samin
Byte pair encoding (BPE) emerges as an effective tokenization method for
tackling the out-of-vocabulary (OOV) challenge in various natural language and
speech processing tasks. Recent research highlights the dependency of BPE
subword tokenization's efficacy on the morphological nature of the language,
particularly in languages rich in inflectional morphology, where fewer BPE
merges suffice for generating highly productive tokens. Motivated by this, our
study empirically identifies the optimal number of BPE tokens for Bengali, a
language known for its morphological complexity, thus enhancing
out-of-distribution automatic speech recognition (ASR) performance.
Experimental evaluation reveals that an excessively high number of BPE tokens
can lead to overfitting, while approximately 500-1000 tokens result in superior
OOV performance. Furthermore, we conduct a comparative analysis of BPE with
character-based and unigram-based tokenization methods. By introducing BPE
tokenization to Bengali ASR, we achieve a substantial reduction in the word
error rate (WER) from 66.44% in our character-based baseline system to 63.80%
on the LB-ASRTD eval set and from 46.34% to 42.80% on the SHRUTI eval set, both
of which include out-of-distribution data.