An Analysis of BPE Vocabulary Trimming in Neural Machine Translation
Marco Cognetta Tatsuya Hiraoka Naoaki Okazaki Rico Sennrich Yuval Pinter
We explore threshold vocabulary trimming in Byte-Pair Encoding subword
tokenization, a postprocessing step that replaces rare subwords with their
component subwords. The technique is available in popular tokenization
libraries but has not been subjected to rigorous scientific scrutiny. While the
removal of rare subwords is suggested as best practice in machine translation
implementations, both as a means to reduce model size and for improving model
performance through robustness, our experiments indicate that, across a large
space of hyperparameter settings, vocabulary trimming fails to improve
performance, and is even prone to incurring heavy degradation.