Meta-Cognitive Analysis: Evaluating Declarative and Procedural Knowledge
  in Datasets and Large Language Models
Zhuoqun Li Hongyu Lin Yaojie Lu Hao Xiang Xianpei Han Le Sun
Declarative knowledge and procedural knowledge are two key parts in
meta-cognitive theory, and these two hold significant importance in
pre-training and inference of LLMs. However, a comprehensive analysis comparing
these two types of knowledge is lacking, primarily due to challenges in
definition, probing and quantitative assessment. In this paper, we explore from
a new perspective by providing ground-truth knowledge for LLMs and evaluating
the effective score. Through extensive experiments with widely-used datasets
and models, we get conclusions: (1) In most tasks, benefits from declarative
knowledge are greater than those from procedural knowledge. (2) Profits of
procedural knowledge are larger than declarative knowledge only in reasoning
tasks with simple logic. (3) As pre-training progresses and size increases,
model ability to utilize both kinds of knowledge significantly improves, but in
different speed. We do detailed analysis for the findings and this can provide
primary guidance for evaluation and enhancement of large language models.