Understanding the Role of Input Token Characters in Language Models: How
  Does Information Loss Affect Performance?
Ahmed Alajrami Katerina Margatina Nikolaos Aletras
Understanding how and what pre-trained language models (PLMs) learn about
language is an open challenge in natural language processing. Previous work has
focused on identifying whether they capture semantic and syntactic information,
and how the data or the pre-training objective affects their performance.
However, to the best of our knowledge, no previous work has specifically
examined how information loss in input token characters affects the performance
of PLMs. In this study, we address this gap by pre-training language models
using small subsets of characters from individual tokens. Surprisingly, we find
that pre-training even under extreme settings, i.e. using only one character of
each token, the performance retention in standard NLU benchmarks and probing
tasks compared to full-token models is high. For instance, a model pre-trained
only on single first characters from tokens achieves performance retention of
approximately $90$\% and $77$\% of the full-token model in SuperGLUE and GLUE
tasks, respectively.