On the Audio Hallucinations in Large Audio-Video Language Models
Taichi Nishimura Shota Nakada Masayoshi Kondo
Large audio-video language models can generate descriptions for both video
and audio. However, they sometimes ignore audio content, producing audio
descriptions solely reliant on visual information. This paper refers to this as
audio hallucinations and analyzes them in large audio-video language models. We
gather 1,000 sentences by inquiring about audio information and annotate them
whether they contain hallucinations. If a sentence is hallucinated, we also
categorize the type of hallucination. The results reveal that 332 sentences are
hallucinated with distinct trends observed in nouns and verbs for each
hallucination type. Based on this, we tackle a task of audio hallucination
classification using pre-trained audio-text models in the zero-shot and
fine-tuning settings. Our experimental results reveal that the zero-shot models
achieve higher performance (52.2% in F1) than the random (40.3%) and the
fine-tuning models achieve 87.9%, outperforming the zero-shot models.