Violation of Expectation via Metacognitive Prompting Reduces Theory of
  Mind Prediction Error in Large Language Models
Courtland Leer Vincent Trost Vineeth Voruganti
Recent research shows that Large Language Models (LLMs) exhibit a compelling
level of proficiency in Theory of Mind (ToM) tasks. This ability to impute
unobservable mental states to others is vital to human social cognition and may
prove equally important in principal-agent relations between individual humans
and Artificial Intelligences (AIs). In this paper, we explore how a mechanism
studied in developmental psychology known as Violation of Expectation (VoE) can
be implemented to reduce errors in LLM prediction about users by leveraging
emergent ToM affordances. And we introduce a \textit{metacognitive prompting}
framework to apply VoE in the context of an AI tutor. By storing and retrieving
facts derived in cases where LLM expectation about the user was violated, we
find that LLMs are able to learn about users in ways that echo theories of
human learning. Finally, we discuss latent hazards and augmentative
opportunities associated with modeling user psychology and propose ways to
mitigate risk along with possible directions for future inquiry.