WYWEB: A NLP Evaluation Benchmark For Classical Chinese
Bo Zhou Qianglong Chen Tianyu Wang Xiaomi Zhong Yin Zhang
To fully evaluate the overall performance of different NLP models in a given
domain, many evaluation benchmarks are proposed, such as GLUE, SuperGLUE and
CLUE. The fi eld of natural language understanding has traditionally focused on
benchmarks for various tasks in languages such as Chinese, English, and
multilingua, however, there has been a lack of attention given to the area of
classical Chinese, also known as "wen yan wen", which has a rich history
spanning thousands of years and holds signifi cant cultural and academic value.
For the prosperity of the NLP community, in this paper, we introduce the WYWEB
evaluation benchmark, which consists of nine NLP tasks in classical Chinese,
implementing sentence classifi cation, sequence labeling, reading
comprehension, and machine translation. We evaluate the existing pre-trained
language models, which are all struggling with this benchmark. We also
introduce a number of supplementary datasets and additional tools to help
facilitate further progress on classical Chinese NLU. The github repository is
https://github.com/baudzhou/WYWEB.