Supporting Anticipatory Governance using LLMs: Evaluating and Aligning
  Large Language Models with the News Media to Anticipate the Negative Impacts
  of AI
Mowafak Allaham Nicholas Diakopoulos
Anticipating the negative impacts of emerging AI technologies is a challenge,
especially in the early stages of development. An understudied approach to such
anticipation is the use of LLMs to enhance and guide this process. Despite
advancements in LLMs and evaluation metrics to account for biases in generated
text, it is unclear how well these models perform in anticipatory tasks.
Specifically, the use of LLMs to anticipate AI impacts raises questions about
the quality and range of categories of negative impacts these models are
capable of generating. In this paper we leverage news media, a diverse data
source that is rich with normative assessments of emerging technologies, to
formulate a taxonomy of impacts to act as a baseline for comparing against. By
computationally analyzing thousands of news articles published by hundreds of
online news domains around the world, we develop a taxonomy consisting of ten
categories of AI impacts. We then evaluate both instruction-based (GPT-4 and
Mistral-7B-Instruct) and fine-tuned completion models (Mistral-7B and GPT-3)
using a sample from this baseline. We find that the generated impacts using
Mistral-7B, fine-tuned on impacts from the news media, tend to be qualitatively
on par with impacts generated using a larger scale model such as GPT-4.
Moreover, we find that these LLMs generate impacts that largely reflect the
taxonomy of negative impacts identified in the news media, however the impacts
produced by instruction-based models had gaps in the production of certain
categories of impacts in comparison to fine-tuned models. This research
highlights a potential bias in state-of-the-art LLMs when used for anticipating
impacts and demonstrates the advantages of aligning smaller LLMs with a diverse
range of impacts, such as those reflected in the news media, to better reflect
such impacts during anticipatory exercises.