Measuring Political Bias in Large Language Models: What Is Said and How
  It Is Said
Yejin Bang Delong Chen Nayeon Lee Pascale Fung
We propose to measure political bias in LLMs by analyzing both the content
and style of their generated content regarding political issues. Existing
benchmarks and measures focus on gender and racial biases. However, political
bias exists in LLMs and can lead to polarization and other harms in downstream
applications. In order to provide transparency to users, we advocate that there
should be fine-grained and explainable measures of political biases generated
by LLMs. Our proposed measure looks at different political issues such as
reproductive rights and climate change, at both the content (the substance of
the generation) and the style (the lexical polarity) of such bias. We measured
the political bias in eleven open-sourced LLMs and showed that our proposed
framework is easily scalable to other topics and is explainable.