Can Large Language Model Comprehend Ancient Chinese? A Preliminary Test
  on ACLUE
Yixuan Zhang Haonan Li
Large language models (LLMs) have showcased remarkable capabilities in
understanding and generating language. However, their ability in comprehending
ancient languages, particularly ancient Chinese, remains largely unexplored. To
bridge this gap, we present ACLUE, an evaluation benchmark designed to assess
the capability of language models in comprehending ancient Chinese. ACLUE
consists of 15 tasks cover a range of skills, spanning phonetic, lexical,
syntactic, semantic, inference and knowledge. Through the evaluation of eight
state-of-the-art LLMs, we observed a noticeable disparity in their performance
between modern Chinese and ancient Chinese. Among the assessed models, ChatGLM2
demonstrates the most remarkable performance, achieving an average score of
37.4%. We have made our code and data public available.