Helmsman of the Masses? Evaluate the Opinion Leadership of Large
  Language Models in the Werewolf Game
Silin Du Xiaowei Zhang
Large language models (LLMs) have exhibited memorable strategic behaviors in
social deductive games. However, the significance of opinion leadership
exhibited by LLM-based agents has been overlooked, which is crucial for
practical applications in multi-agent and human-AI interaction settings.
Opinion leaders are individuals who have a noticeable impact on the beliefs and
behaviors of others within a social group. In this work, we employ the Werewolf
game as a simulation platform to assess the opinion leadership of LLMs. The
game features the role of the Sheriff, tasked with summarizing arguments and
recommending decision options, and therefore serves as a credible proxy for an
opinion leader. We develop a framework integrating the Sheriff role and devise
two novel metrics for evaluation based on the critical characteristics of
opinion leaders. The first metric measures the reliability of the opinion
leader, and the second assesses the influence of the opinion leader on other
players' decisions. We conduct extensive experiments to evaluate LLMs of
different scales. In addition, we collect a Werewolf question-answering dataset
(WWQA) to assess and enhance LLM's grasp of the game rules, and we also
incorporate human participants for further analysis. The results suggest that
the Werewolf game is a suitable test bed to evaluate the opinion leadership of
LLMs and few LLMs possess the capacity for opinion leadership.