SALMA: Arabic Sense-Annotated Corpus and WSD Benchmarks
Mustafa Jarrar Sanad Malaysha Tymaa Hammouda Mohammed Khalilia
SALMA, the first Arabic sense-annotated corpus, consists of ~34K tokens,
which are all sense-annotated. The corpus is annotated using two different
sense inventories simultaneously (Modern and Ghani). SALMA novelty lies in how
tokens and senses are associated. Instead of linking a token to only one
intended sense, SALMA links a token to multiple senses and provides a score to
each sense. A smart web-based annotation tool was developed to support scoring
multiple senses against a given word. In addition to sense annotations, we also
annotated the corpus using six types of named entities. The quality of our
annotations was assessed using various metrics (Kappa, Linear Weighted Kappa,
Quadratic Weighted Kappa, Mean Average Error, and Root Mean Square Error),
which show very high inter-annotator agreement. To establish a Word Sense
Disambiguation baseline using our SALMA corpus, we developed an end-to-end Word
Sense Disambiguation system using Target Sense Verification. We used this
system to evaluate three Target Sense Verification models available in the
literature. Our best model achieved an accuracy with 84.2% using Modern and
78.7% using Ghani. The full corpus and the annotation tool are open-source and
publicly available at https://sina.birzeit.edu/salma/.