On the Decision-Making Abilities in Role-Playing using Large Language
  Models
Chenglei Shen Guofu Xie Xiao Zhang Jun Xu
Large language models (LLMs) are now increasingly utilized for role-playing
tasks, especially in impersonating domain-specific experts, primarily through
role-playing prompts. When interacting in real-world scenarios, the
decision-making abilities of a role significantly shape its behavioral
patterns. In this paper, we concentrate on evaluating the decision-making
abilities of LLMs post role-playing thereby validating the efficacy of
role-playing. Our goal is to provide metrics and guidance for enhancing the
decision-making abilities of LLMs in role-playing tasks. Specifically, we first
use LLMs to generate virtual role descriptions corresponding to the 16
personality types of Myers-Briggs Type Indicator (abbreviated as MBTI)
representing a segmentation of the population. Then we design specific
quantitative operations to evaluate the decision-making abilities of LLMs post
role-playing from four aspects: adaptability, exploration$\&$exploitation
trade-off ability, reasoning ability, and safety. Finally, we analyze the
association between the performance of decision-making and the corresponding
MBTI types through GPT-4. Extensive experiments demonstrate stable differences
in the four aspects of decision-making abilities across distinct roles,
signifying a robust correlation between decision-making abilities and the roles
emulated by LLMs. These results underscore that LLMs can effectively
impersonate varied roles while embodying their genuine sociological
characteristics.