Quantifying the Persona Effect in LLM Simulations
Tiancheng Hu Nigel Collier
Large language models (LLMs) have shown remarkable promise in simulating
human language use and behavior. In this study, we delve into the intersection
of persona variables and the capability of LLMs to simulate different
perspectives. We find that persona variables can explain <10\% variance in
annotations in existing subjective NLP datasets. Nonetheless, incorporating
them via prompting in LLMs provides modest improvement. Persona prompting is
most effective on data samples where disagreements among annotators are
frequent yet confined to a limited range. A linear correlation exists: the more
persona variables influence human annotations, the better LLMs predictions are
using persona prompting. However, when the utility of persona variables is low
(i.e., explaining <10\% of human annotations), persona prompting has little
effect. Most subjective NLP datasets fall into this category, casting doubt on
simulating diverse perspectives in the current NLP landscape.