Studying and improving reasoning in humans and machines
Nicolas Yax Hernan Anlló Stefano Palminteri
In the present study, we investigate and compare reasoning in large language
models (LLM) and humans using a selection of cognitive psychology tools
traditionally dedicated to the study of (bounded) rationality. To do so, we
presented to human participants and an array of pretrained LLMs new variants of
classical cognitive experiments, and cross-compared their performances. Our
results showed that most of the included models presented reasoning errors akin
to those frequently ascribed to error-prone, heuristic-based human reasoning.
Notwithstanding this superficial similarity, an in-depth comparison between
humans and LLMs indicated important differences with human-like reasoning, with
models limitations disappearing almost entirely in more recent LLMs releases.
Moreover, we show that while it is possible to devise strategies to induce
better performance, humans and machines are not equally-responsive to the same
prompting schemes. We conclude by discussing the epistemological implications
and challenges of comparing human and machine behavior for both artificial
intelligence and cognitive psychology.