CIF-Bench: A Chinese Instruction-Following Benchmark for Evaluating the
  Generalizability of Large Language Models
Yizhi LI Ge Zhang Xingwei Qu Jiali Li Zhaoqun Li Zekun Wang Hao Li Ruibin Yuan Yinghao Ma Kai Zhang Wangchunshu Zhou Yiming Liang Lei Zhang Lei Ma Jiajun Zhang Zuowen Li Stephen W. Huang Chenghua Lin Wenhu Chen Jie Fu
The advancement of large language models (LLMs) has enhanced the ability to
generalize across a wide range of unseen natural language processing (NLP)
tasks through instruction-following. Yet, their effectiveness often diminishes
in low-resource languages like Chinese, exacerbated by biased evaluations from
data leakage, casting doubt on their true generalizability to new linguistic
territories. In response, we introduce the Chinese Instruction-Following
Benchmark (CIF-Bench), designed to evaluate the zero-shot generalizability of
LLMs to the Chinese language. CIF-Bench comprises 150 tasks and 15,000
input-output pairs, developed by native speakers to test complex reasoning and
Chinese cultural nuances across 20 categories. To mitigate evaluation bias, we
release only half of the dataset publicly, with the remainder kept private, and
introduce diversified instructions to minimize score variance, totaling 45,000
data instances. Our evaluation of 28 selected LLMs reveals a noticeable
performance gap, with the best model scoring only 52.9%, highlighting the
limitations of LLMs in less familiar language and task contexts. This work aims
to uncover the current limitations of LLMs in handling Chinese tasks, pushing
towards the development of more culturally informed and linguistically diverse
models with the released data and benchmark
(https://yizhilll.github.io/CIF-Bench/).