Beyond prompt brittleness: Evaluating the reliability and consistency of
  political worldviews in LLMs
Tanise Ceron Neele Falk Ana Barić Dmitry Nikolaev Sebastian Padó
Due to the widespread use of large language models (LLMs) in ubiquitous
systems, we need to understand whether they embed a specific worldview and what
these views reflect. Recent studies report that, prompted with political
questionnaires, LLMs show left-liberal leanings. However, it is as yet unclear
whether these leanings are reliable (robust to prompt variations) and whether
the leaning is consistent across policies and political leaning. We propose a
series of tests which assess the reliability and consistency of LLMs' stances
on political statements based on a dataset of voting-advice questionnaires
collected from seven EU countries and annotated for policy domains. We study
LLMs ranging in size from 7B to 70B parameters and find that their reliability
increases with parameter count. Larger models show overall stronger alignment
with left-leaning parties but differ among policy programs: They evince a
(left-wing) positive stance towards environment protection, social welfare but
also (right-wing) law and order, with no consistent preferences in foreign
policy, migration, and economy.