Word Boundary Information Isn't Useful for Encoder Language Models
Edward Gow-Smith Dylan Phelps Harish Tayyar Madabushi Carolina Scarton Aline Villavicencio
All existing transformer-based approaches to NLP using subword tokenisation
algorithms encode whitespace (word boundary information) through the use of
special space symbols (such as \#\# or \_) forming part of tokens. These
symbols have been shown to a) lead to reduced morphological validity of
tokenisations, and b) give substantial vocabulary redundancy. As such, removing
these symbols has been shown to have a beneficial effect on the processing of
morphologically complex words for transformer encoders in the pretrain-finetune
paradigm. In this work, we explore whether word boundary information is at all
useful to such models. In particular, we train transformer encoders across four
different training scales, and investigate several alternative approaches to
including word boundary information, evaluating on a range of tasks across
different domains and problem set-ups: GLUE (for sentence-level
classification), NER (for token-level classification), and two classification
datasets involving complex words (Superbizarre and FLOTA). Overall, through an
extensive experimental setup that includes the pre-training of 29 models, we
find no substantial improvements from our alternative approaches, suggesting
that modifying tokenisers to remove word boundary information isn't leading to
a loss of useful information.