Automatic Hallucination Assessment for Aligned Large Language Models via
  Transferable Adversarial Attacks
Xiaodong Yu Hao Cheng Xiaodong Liu Dan Roth Jianfeng Gao
Although remarkable progress has been achieved in preventing large language
model (LLM) hallucinations using instruction tuning and retrieval augmentation,
it remains challenging to measure the reliability of LLMs using human-crafted
evaluation data which is not available for many tasks and domains and could
suffer from data leakage. Inspired by adversarial machine learning, this paper
aims to develop a method of automatically generating evaluation data by
appropriately modifying existing data on which LLMs behave faithfully.
Specifically, this paper presents AutoDebug, an LLM-based framework to use
prompting chaining to generate transferable adversarial attacks in the form of
question-answering examples. We seek to understand the extent to which these
examples trigger the hallucination behaviors of LLMs.
  We implement AutoDebug using ChatGPT and evaluate the resulting two variants
of a popular open-domain question-answering dataset, Natural Questions (NQ), on
a collection of open-source and proprietary LLMs under various prompting
settings. Our generated evaluation data is human-readable and, as we show,
humans can answer these modified questions well. Nevertheless, we observe
pronounced accuracy drops across multiple LLMs including GPT-4. Our
experimental results show that LLMs are likely to hallucinate in two categories
of question-answering scenarios where (1) there are conflicts between knowledge
given in the prompt and their parametric knowledge, or (2) the knowledge
expressed in the prompt is complex. Finally, we find that the adversarial
examples generated by our method are transferable across all considered LLMs.
The examples generated by a small model can be used to debug a much larger
model, making our approach cost-effective.