Comparing Hallucination Detection Metrics for Multilingual Generation
Haoqiang Kang Terra Blevins Luke Zettlemoyer
While many automatic hallucination detection techniques have been proposed
for English texts, their effectiveness in multilingual contexts remains
unexplored. This paper aims to bridge the gap in understanding how these
hallucination detection metrics perform on non-English languages. We evaluate
the efficacy of various detection metrics, including lexical metrics like ROUGE
and Named Entity Overlap and Natural Language Inference (NLI)-based metrics, at
detecting hallucinations in biographical summaries in many languages; we also
evaluate how correlated these different metrics are to gauge whether they
measure the same phenomena. Our empirical analysis reveals that while lexical
metrics show limited effectiveness, NLI-based metrics perform well in
high-resource languages at the sentence level. In contrast, NLI-based metrics
often fail to detect atomic fact hallucinations. Our findings highlight
existing gaps in multilingual hallucination detection and motivate future
research to develop more robust detection methods for LLM hallucination in
other languages.