An Improved Traditional Chinese Evaluation Suite for Foundation Model
Zhi-Rui Tam Ya-Ting Pai Yen-Wei Lee Sega Cheng Hong-Han Shuai
We present TMMLU+, a comprehensive dataset designed for the Traditional
Chinese massive multitask language understanding dataset. TMMLU+ is a
multiple-choice question-answering dataset with 66 subjects from elementary to
professional level. Compared to its predecessor, TMMLU, TMMLU+ is six times
larger and boasts a more balanced subject distribution. We included benchmark
results in TMMLU+ from closed-source models and 24 open-weight Chinese large
language models of parameters ranging from 1.8B to 72B. Our findings reveal
that Traditional Chinese models still trail behind their Simplified Chinese
counterparts. Additionally, current large language models have yet to
outperform human performance in average scores. We publicly release our dataset
and the corresponding benchmark source code.