Theory of Mind in Large Language Models: Examining Performance of 11
  State-of-the-Art models vs. Children Aged 7-10 on Advanced Tests
Max J. van Duijn Bram M. A. van Dijk Tom Kouwenhoven Werner de Valk Marco R. Spruit Peter van der Putten
To what degree should we ascribe cognitive capacities to Large Language
Models (LLMs), such as the ability to reason about intentions and beliefs known
as Theory of Mind (ToM)? Here we add to this emerging debate by (i) testing 11
base- and instruction-tuned LLMs on capabilities relevant to ToM beyond the
dominant false-belief paradigm, including non-literal language usage and
recursive intentionality; (ii) using newly rewritten versions of standardized
tests to gauge LLMs' robustness; (iii) prompting and scoring for open besides
closed questions; and (iv) benchmarking LLM performance against that of
children aged 7-10 on the same tasks. We find that instruction-tuned LLMs from
the GPT family outperform other models, and often also children. Base-LLMs are
mostly unable to solve ToM tasks, even with specialized prompting. We suggest
that the interlinked evolution and development of language and ToM may help
explain what instruction-tuning adds: rewarding cooperative communication that
takes into account interlocutor and context. We conclude by arguing for a
nuanced perspective on ToM in LLMs.