Cross-Lingual NER for Financial Transaction Data in Low-Resource
  Languages
Sunisth Kumar Davide Liu Alexandre Boulenger
We propose an efficient modeling framework for cross-lingual named entity
recognition in semi-structured text data. Our approach relies on both knowledge
distillation and consistency training. The modeling framework leverages
knowledge from a large language model (XLMRoBERTa) pre-trained on the source
language, with a student-teacher relationship (knowledge distillation). The
student model incorporates unsupervised consistency training (with KL
divergence loss) on the low-resource target language.
  We employ two independent datasets of SMSs in English and Arabic, each
carrying semi-structured banking transaction information, and focus on
exhibiting the transfer of knowledge from English to Arabic. With access to
only 30 labeled samples, our model can generalize the recognition of merchants,
amounts, and other fields from English to Arabic. We show that our modeling
approach, while efficient, performs best overall when compared to
state-of-the-art approaches like DistilBERT pre-trained on the target language
or a supervised model directly trained on labeled data in the target language.
  Our experiments show that it is enough to learn to recognize entities in
English to reach reasonable performance in a low-resource language in the
presence of a few labeled samples of semi-structured data. The proposed
framework has implications for developing multi-lingual applications,
especially in geographies where digital endeavors rely on both English and one
or more low-resource language(s), sometimes mixed with English or employed
singly.