To token or not to token: A Comparative Study of Text Representations
  for Cross-Lingual Transfer
Md Mushfiqur Rahman Fardin Ahsan Sakib Fahim Faisal Antonios Anastasopoulos
Choosing an appropriate tokenization scheme is often a bottleneck in
low-resource cross-lingual transfer. To understand the downstream implications
of text representation choices, we perform a comparative analysis on language
models having diverse text representation modalities including 2
segmentation-based models (\texttt{BERT}, \texttt{mBERT}), 1 image-based model
(\texttt{PIXEL}), and 1 character-level model (\texttt{CANINE}). First, we
propose a scoring Language Quotient (LQ) metric capable of providing a weighted
representation of both zero-shot and few-shot evaluation combined. Utilizing
this metric, we perform experiments comprising 19 source languages and 133
target languages on three tasks (POS tagging, Dependency parsing, and NER). Our
analysis reveals that image-based models excel in cross-lingual transfer when
languages are closely related and share visually similar scripts. However, for
tasks biased toward word meaning (POS, NER), segmentation-based models prove to
be superior. Furthermore, in dependency parsing tasks where word relationships
play a crucial role, models with their character-level focus, outperform
others. Finally, we propose a recommendation scheme based on our findings to
guide model selection according to task and language requirements.