Measuring and Reducing LLM Hallucination without Gold-Standard Answers
  via Expertise-Weighting
Jiaheng Wei Yuanshun Yao Jean-Francois Ton Hongyi Guo Andrew Estornell Yang Liu
LLM hallucination, i.e. generating factually incorrect yet seemingly
convincing answers, is currently a major threat to the trustworthiness and
reliability of LLMs. The first step towards solving this complicated problem is
to measure it. However, existing hallucination metrics require to have a
benchmark dataset with gold-standard answers, i.e. "best" or "correct" answers
written by humans. Such requirement makes hallucination measurement costly and
prone to human errors. In this work, we propose Factualness Evaluations via
Weighting LLMs (FEWL), the first hallucination metric that is specifically
designed for the scenario when gold-standard answers are absent. FEWL leverages
the answers from off-the-shelf LLMs that serve as a proxy of gold-standard
answers. The key challenge is how to quantify the expertise of reference LLMs
resourcefully. We show FEWL has certain theoretical guarantees and demonstrate
empirically it gives more accurate hallucination measures than naively using
reference LLMs. We also show how to leverage FEWL to reduce hallucination
through both in-context learning and supervised finetuning. Last, we build a
large-scale benchmark dataset to facilitate LLM hallucination research.