Greed is All You Need: An Evaluation of Tokenizer Inference Methods
Omri Uzan Craig W. Schmidt Chris Tanner Yuval Pinter
While subword tokenizers such as BPE and WordPiece are typically used to
build vocabularies for NLP models, the method of decoding text into a sequence
of tokens from these vocabularies is often left unspecified, or ill-suited to
the method in which they were constructed. We provide a controlled analysis of
seven tokenizer inference methods across four different algorithms and three
vocabulary sizes, performed on a novel intrinsic evaluation suite we curated
for English, combining measures rooted in morphology, cognition, and
information theory. We show that for the most commonly used tokenizers, greedy
inference performs surprisingly well; and that SaGe, a recently-introduced
contextually-informed tokenizer, outperforms all others on morphological
alignment.