HALC: Object Hallucination Reduction via Adaptive Focal-Contrast
  Decoding
Zhaorun Chen Zhuokai Zhao Hongyin Luo Huaxiu Yao Bo Li Jiawei Zhou
While large vision-language models (LVLMs) have demonstrated impressive
capabilities in interpreting multi-modal contexts, they invariably suffer from
object hallucinations (OH). We introduce HALC, a novel decoding algorithm
designed to mitigate OH in LVLMs. HALC leverages distinct fine-grained optimal
visual information in vision-language tasks and operates on both local and
global contexts simultaneously. Specifically, HALC integrates a robust
auto-focal grounding mechanism (locally) to correct hallucinated tokens on the
fly, and a specialized beam search algorithm (globally) to significantly reduce
OH while preserving text generation quality. Additionally, HALC can be
integrated into any LVLMs as a plug-and-play module without extra training.
Extensive experimental studies demonstrate the effectiveness of HALC in
reducing OH, outperforming state-of-the-arts across four benchmarks.