Risk and Response in Large Language Models: Evaluating Key Threat
  Categories
Bahareh Harandizadeh Abel Salinas Fred Morstatter
This paper explores the pressing issue of risk assessment in Large Language
Models (LLMs) as they become increasingly prevalent in various applications.
Focusing on how reward models, which are designed to fine-tune pretrained LLMs
to align with human values, perceive and categorize different types of risks,
we delve into the challenges posed by the subjective nature of preference-based
training data. By utilizing the Anthropic Red-team dataset, we analyze major
risk categories, including Information Hazards, Malicious Uses, and
Discrimination/Hateful content. Our findings indicate that LLMs tend to
consider Information Hazards less harmful, a finding confirmed by a specially
developed regression model. Additionally, our analysis shows that LLMs respond
less stringently to Information Hazards compared to other risks. The study
further reveals a significant vulnerability of LLMs to jailbreaking attacks in
Information Hazard scenarios, highlighting a critical security concern in LLM
risk assessment and emphasizing the need for improved AI safety measures.