Violet: A Vision-Language Model for Arabic Image Captioning with Gemini
  Decoder
Abdelrahman Mohamed Fakhraddin Alwajih El Moatez Billah Nagoudi Alcides Alcoba Inciarte Muhammad Abdul-Mageed
Although image captioning has a vast array of applications, it has not
reached its full potential in languages other than English. Arabic, for
instance, although the native language of more than 400 million people, remains
largely underrepresented in this area. This is due to the lack of labeled data
and powerful Arabic generative models. We alleviate this issue by presenting a
novel vision-language model dedicated to Arabic, dubbed \textit{Violet}. Our
model is based on a vision encoder and a Gemini text decoder that maintains
generation fluency while allowing fusion between the vision and language
components. To train our model, we introduce a new method for automatically
acquiring data from available English datasets. We also manually prepare a new
dataset for evaluation. \textit{Violet} performs sizeably better than our
baselines on all of our evaluation datasets. For example, it reaches a CIDEr
score of $61.2$ on our manually annotated dataset and achieves an improvement
of $13$ points on Flickr8k.