Assessing LLMs for Moral Value Pluralism
Noam Benkler Drisana Mosaphir Scott Friedman Andrew Smart Sonja Schmer-Galunder
The fields of AI current lacks methods to quantitatively assess and
potentially alter the moral values inherent in the output of large language
models (LLMs). However, decades of social science research has developed and
refined widely-accepted moral value surveys, such as the World Values Survey
(WVS), eliciting value judgments from direct questions in various geographies.
We have turned those questions into value statements and use NLP to compute to
how well popular LLMs are aligned with moral values for various demographics
and cultures. While the WVS is accepted as an explicit assessment of values, we
lack methods for assessing implicit moral and cultural values in media, e.g.,
encountered in social media, political rhetoric, narratives, and generated by
AI systems such as LLMs that are increasingly present in our daily lives. As we
consume online content and utilize LLM outputs, we might ask, which moral
values are being implicitly promoted or undercut, or -- in the case of LLMs --
if they are intending to represent a cultural identity, are they doing so
consistently? In this paper we utilize a Recognizing Value Resonance (RVR) NLP
model to identify WVS values that resonate and conflict with a given passage of
output text. We apply RVR to the text generated by LLMs to characterize
implicit moral values, allowing us to quantify the moral/cultural distance
between LLMs and various demographics that have been surveyed using the WVS. In
line with other work we find that LLMs exhibit several Western-centric value
biases; they overestimate how conservative people in non-Western countries are,
they are less accurate in representing gender for non-Western countries, and
portray older populations as having more traditional values. Our results
highlight value misalignment and age groups, and a need for social science
informed technological solutions addressing value plurality in LLMs.