Reading Users' Minds from What They Say: An Investigation into LLM-based
  Empathic Mental Inference
Qihao Zhu Leah Chong Maria Yang Jianxi Luo
In human-centered design, developing a comprehensive and in-depth
understanding of user experiences, i.e., empathic understanding, is paramount
for designing products that truly meet human needs. Nevertheless, accurately
comprehending the real underlying mental states of a large human population
remains a significant challenge today. This difficulty mainly arises from the
trade-off between depth and scale of user experience research: gaining in-depth
insights from a small group of users does not easily scale to a larger
population, and vice versa. This paper investigates the use of Large Language
Models (LLMs) for performing mental inference tasks, specifically inferring
users' underlying goals and fundamental psychological needs (FPNs). Baseline
and benchmark datasets were collected from human users and designers to develop
an empathic accuracy metric for measuring the mental inference performance of
LLMs. The empathic accuracy of inferring goals and FPNs of different LLMs with
varied zero-shot prompt engineering techniques are experimented against that of
human designers. Experimental results suggest that LLMs can infer and
understand the underlying goals and FPNs of users with performance comparable
to that of human designers, suggesting a promising avenue for enhancing the
scalability of empathic design approaches through the integration of advanced
artificial intelligence technologies. This work has the potential to
significantly augment the toolkit available to designers during human-centered
design, enabling the development of both large-scale and in-depth understanding
of users' experiences.