Benchmarking Large Language Models for Persian: A Preliminary Study
  Focusing on ChatGPT
Amirhossein Abaskohi Sara Baruni Mostafa Masoudi Nesa Abbasi Mohammad Hadi Babalou Ali Edalat Sepehr Kamahi Samin Mahdizadeh Sani Nikoo Naghavian Danial Namazifard Pouya Sadeghi Yadollah Yaghoobzadeh
This paper explores the efficacy of large language models (LLMs) for Persian.
While ChatGPT and consequent LLMs have shown remarkable performance in English,
their efficiency for more low-resource languages remains an open question. We
present the first comprehensive benchmarking study of LLMs across diverse
Persian language tasks. Our primary focus is on GPT-3.5-turbo, but we also
include GPT-4 and OpenChat-3.5 to provide a more holistic evaluation. Our
assessment encompasses a diverse set of tasks categorized into classic,
reasoning, and knowledge-based domains. To enable a thorough comparison, we
evaluate LLMs against existing task-specific fine-tuned models. Given the
limited availability of Persian datasets for reasoning tasks, we introduce two
new benchmarks: one based on elementary school math questions and another
derived from the entrance exams for 7th and 10th grades. Our findings reveal
that while LLMs, especially GPT-4, excel in tasks requiring reasoning abilities
and a broad understanding of general knowledge, they often lag behind smaller
pre-trained models fine-tuned specifically for particular tasks. Additionally,
we observe improved performance when test sets are translated to English before
inputting them into GPT-3.5. These results highlight the significant potential
for enhancing LLM performance in the Persian language. This is particularly
noteworthy due to the unique attributes of Persian, including its distinct
alphabet and writing styles.