CORECODE: A Common Sense Annotated Dialogue Dataset with Benchmark Tasks
  for Chinese Large Language Models
Dan Shi Chaobin You Jiantao Huang Taihao Li Deyi Xiong
As an indispensable ingredient of intelligence, commonsense reasoning is
crucial for large language models (LLMs) in real-world scenarios. In this
paper, we propose CORECODE, a dataset that contains abundant commonsense
knowledge manually annotated on dyadic dialogues, to evaluate the commonsense
reasoning and commonsense conflict detection capabilities of Chinese LLMs. We
categorize commonsense knowledge in everyday conversations into three
dimensions: entity, event, and social interaction. For easy and consistent
annotation, we standardize the form of commonsense knowledge annotation in
open-domain dialogues as "domain: slot = value". A total of 9 domains and 37
slots are defined to capture diverse commonsense knowledge. With these
pre-defined domains and slots, we collect 76,787 commonsense knowledge
annotations from 19,700 dialogues through crowdsourcing. To evaluate and
enhance the commonsense reasoning capability for LLMs on the curated dataset,
we establish a series of dialogue-level reasoning and detection tasks,
including commonsense knowledge filling, commonsense knowledge generation,
commonsense conflict phrase detection, domain identification, slot
identification, and event causal inference. A wide variety of existing
open-source Chinese LLMs are evaluated with these tasks on our dataset.
Experimental results demonstrate that these models are not competent to predict
CORECODE's plentiful reasoning content, and even ChatGPT could only achieve
0.275 and 0.084 accuracy on the domain identification and slot identification
tasks under the zero-shot setting. We release the data and codes of CORECODE at
https://github.com/danshi777/CORECODE to promote commonsense reasoning
evaluation and study of LLMs in the context of daily conversations.