What is the best recipe for character-level encoder-only modelling?
Kris Cao
This paper aims to benchmark recent progress in language understanding models
that output contextualised representations at the character level. Many such
modelling architectures and methods to train those architectures have been
proposed, but it is currently unclear what the relative contributions of the
architecture vs. the pretraining objective are to final model performance. We
explore the design space of such models, comparing architectural innovations
and a variety of different pretraining objectives on a suite of evaluation
tasks with a fixed training procedure in order to find the currently optimal
way to build and train character-level BERT-like models. We find that our best
performing character-level model exceeds the performance of a token-based model
trained with the same settings on the same data, suggesting that
character-level models are ready for more widespread adoption. Unfortunately,
the best method to train character-level models still relies on a subword-level
tokeniser during pretraining, and final model performance is highly dependent
on tokeniser quality. We believe our results demonstrate the readiness of
character-level models for multilingual language representation, and encourage
NLP practitioners to try them as drop-in replacements for token-based models.