Identifying Multiple Personalities in Large Language Models with
  External Evaluation
Xiaoyang Song Yuta Adachi Jessie Feng Mouwei Lin Linhao Yu Frank Li Akshat Gupta Gopala Anumanchipalli Simerjot Kaur
As Large Language Models (LLMs) are integrated with human daily applications
rapidly, many societal and ethical concerns are raised regarding the behavior
of LLMs. One of the ways to comprehend LLMs' behavior is to analyze their
personalities. Many recent studies quantify LLMs' personalities using
self-assessment tests that are created for humans. Yet many critiques question
the applicability and reliability of these self-assessment tests when applied
to LLMs. In this paper, we investigate LLM personalities using an alternate
personality measurement method, which we refer to as the external evaluation
method, where instead of prompting LLMs with multiple-choice questions in the
Likert scale, we evaluate LLMs' personalities by analyzing their responses
toward open-ended situational questions using an external machine learning
model. We first fine-tuned a Llama2-7B model as the MBTI personality predictor
that outperforms the state-of-the-art models as the tool to analyze LLMs'
responses. Then, we prompt the LLMs with situational questions and ask them to
generate Twitter posts and comments, respectively, in order to assess their
personalities when playing two different roles. Using the external personality
evaluation method, we identify that the obtained personality types for LLMs are
significantly different when generating posts versus comments, whereas humans
show a consistent personality profile in these two different situations. This
shows that LLMs can exhibit different personalities based on different
scenarios, thus highlighting a fundamental difference between personality in
LLMs and humans. With our work, we call for a re-evaluation of personality
definition and measurement in LLMs.