Leveraging Closed-Access Multilingual Embedding for Automatic Sentence
  Alignment in Low Resource Languages
Idris Abdulmumin Auwal Abubakar Khalid Shamsuddeen Hassan Muhammad Ibrahim Said Ahmad Lukman Jibril Aliyu Babangida Sani Bala Mairiga Abduljalil Sani Ahmad Hassan
The importance of qualitative parallel data in machine translation has long
been determined but it has always been very difficult to obtain such in
sufficient quantity for the majority of world languages, mainly because of the
associated cost and also the lack of accessibility to these languages. Despite
the potential for obtaining parallel datasets from online articles using
automatic approaches, forensic investigations have found a lot of
quality-related issues such as misalignment, and wrong language codes. In this
work, we present a simple but qualitative parallel sentence aligner that
carefully leveraged the closed-access Cohere multilingual embedding, a solution
that ranked second in the just concluded #CoHereAIHack 2023 Challenge (see
https://ai6lagos.devpost.com). The proposed approach achieved $94.96$ and
$54.83$ f1 scores on FLORES and MAFAND-MT, compared to $3.64$ and $0.64$ of
LASER respectively. Our method also achieved an improvement of more than 5 BLEU
scores over LASER, when the resulting datasets were used with MAFAND-MT dataset
to train translation models. Our code and data are available for research
purposes here (https://github.com/abumafrim/Cohere-Align).