Do Large Language Models Show Decision Heuristics Similar to Humans? A
  Case Study Using GPT-3.5
Gaurav Suri Lily R. Slater Ali Ziaee Morgan Nguyen
A Large Language Model (LLM) is an artificial intelligence system that has
been trained on vast amounts of natural language data, enabling it to generate
human-like responses to written or spoken language input. GPT-3.5 is an example
of an LLM that supports a conversational agent called ChatGPT. In this work, we
used a series of novel prompts to determine whether ChatGPT shows heuristics,
biases, and other decision effects. We also tested the same prompts on human
participants. Across four studies, we found that ChatGPT was influenced by
random anchors in making estimates (Anchoring Heuristic, Study 1); it judged
the likelihood of two events occurring together to be higher than the
likelihood of either event occurring alone, and it was erroneously influenced
by salient anecdotal information (Representativeness and Availability
Heuristic, Study 2); it found an item to be more efficacious when its features
were presented positively rather than negatively - even though both
presentations contained identical information (Framing Effect, Study 3); and it
valued an owned item more than a newly found item even though the two items
were identical (Endowment Effect, Study 4). In each study, human participants
showed similar effects. Heuristics and related decision effects in humans are
thought to be driven by cognitive and affective processes such as loss aversion
and effort reduction. The fact that an LLM - which lacks these processes - also
shows such effects invites consideration of the possibility that language may
play a role in generating these effects in humans.