Cross-Lingual Knowledge Distillation for Answer Sentence Selection in
  Low-Resource Languages
Shivanshu Gupta Yoshitomo Matsubara Ankit Chadha Alessandro Moschitti
While impressive performance has been achieved on the task of Answer Sentence
Selection (AS2) for English, the same does not hold for languages that lack
large labeled datasets. In this work, we propose Cross-Lingual Knowledge
Distillation (CLKD) from a strong English AS2 teacher as a method to train AS2
models for low-resource languages in the tasks without the need of labeled data
for the target language. To evaluate our method, we introduce 1) Xtr-WikiQA, a
translation-based WikiQA dataset for 9 additional languages, and 2) TyDi-AS2, a
multilingual AS2 dataset with over 70K questions spanning 8 typologically
diverse languages. We conduct extensive experiments on Xtr-WikiQA and TyDi-AS2
with multiple teachers, diverse monolingual and multilingual pretrained
language models (PLMs) as students, and both monolingual and multilingual
training. The results demonstrate that CLKD either outperforms or rivals even
supervised fine-tuning with the same amount of labeled data and a combination
of machine translation and the teacher model. Our method can potentially enable
stronger AS2 models for low-resource languages, while TyDi-AS2 can serve as the
largest multilingual AS2 dataset for further studies in the research community.