Comparing Inferential Strategies of Humans and Large Language Models in
  Deductive Reasoning
Philipp Mondorf Barbara Plank
Deductive reasoning plays a pivotal role in the formulation of sound and
cohesive arguments. It allows individuals to draw conclusions that logically
follow, given the truth value of the information provided. Recent progress in
the domain of large language models (LLMs) has showcased their capability in
executing deductive reasoning tasks. Nonetheless, a significant portion of
research primarily assesses the accuracy of LLMs in solving such tasks, often
overlooking a deeper analysis of their reasoning behavior. In this study, we
draw upon principles from cognitive psychology to examine inferential
strategies employed by LLMs, through a detailed evaluation of their responses
to propositional logic problems. Our findings indicate that LLMs display
reasoning patterns akin to those observed in humans, including strategies like
$\textit{supposition following}$ or $\textit{chain construction}$. Moreover,
our research demonstrates that the architecture and scale of the model
significantly affect its preferred method of reasoning, with more advanced
models tending to adopt strategies more frequently than less sophisticated
ones. Importantly, we assert that a model's accuracy, that is the correctness
of its final conclusion, does not necessarily reflect the validity of its
reasoning process. This distinction underscores the necessity for more nuanced
evaluation procedures in the field.