AraMUS: Pushing the Limits of Data and Model Scale for Arabic Natural
  Language Processing
Asaad Alghamdi Xinyu Duan Wei Jiang Zhenhai Wang Yimeng Wu Qingrong Xia Zhefeng Wang Yi Zheng Mehdi Rezagholizadeh Baoxing Huai Peilun Cheng Abbas Ghaddar
Developing monolingual large Pre-trained Language Models (PLMs) is shown to
be very successful in handling different tasks in Natural Language Processing
(NLP). In this work, we present AraMUS, the largest Arabic PLM with 11B
parameters trained on 529GB of high-quality Arabic textual data. AraMUS
achieves state-of-the-art performances on a diverse set of Arabic
classification and generative tasks. Moreover, AraMUS shows impressive few-shot
learning abilities compared with the best existing Arabic PLMs.