Systematic Biases in LLM Simulations of Debates
Amir Taubenfeld Yaniv Dover Roi Reichart Ariel Goldstein
Recent advancements in natural language processing, especially the emergence
of Large Language Models (LLMs), have opened exciting possibilities for
constructing computational simulations designed to replicate human behavior
accurately. However, LLMs are complex statistical learners without
straightforward deductive rules, making them prone to unexpected behaviors. In
this study, we highlight the limitations of LLMs in simulating human
interactions, particularly focusing on LLMs' ability to simulate political
debates. Our findings indicate a tendency for LLM agents to conform to the
model's inherent social biases despite being directed to debate from certain
political perspectives. This tendency results in behavioral patterns that seem
to deviate from well-established social dynamics among humans. We reinforce
these observations using an automatic self-fine-tuning method, which enables us
to manipulate the biases within the LLM and demonstrate that agents
subsequently align with the altered biases. These results underscore the need
for further research to develop methods that help agents overcome these biases,
a critical step toward creating more realistic simulations.