AVicuna: Audio-Visual LLM with Interleaver and Context-Boundary
  Alignment for Temporal Referential Dialogue
Yunlong Tang Daiki Shimada Jing Bi Chenliang Xu
In everyday communication, humans frequently use speech and gestures to refer
to specific areas or objects, a process known as Referential Dialogue (RD).
While prior studies have investigated RD through Large Language Models (LLMs)
or Large Multimodal Models (LMMs) in static contexts, the exploration of
Temporal Referential Dialogue (TRD) within audio-visual media remains limited.
Two primary challenges hinder progress in this field: (1) the absence of
comprehensive, untrimmed audio-visual video datasets with precise temporal
annotations, and (2) the need for methods to integrate complex temporal
auditory and visual cues effectively. To address these challenges, we introduce
a novel framework to generate PU-VALOR, an extensive audio-visual dataset
comprising over 114,000 untrimmed videos with accurate temporal demarcations.
We also present AVicuna, featuring an Audio-Visual Tokens Interleaver (AVTI)
that ensures the temporal alignment of audio-visual information. Additionally,
we develop the A5-222K dataset, encompassing more than 200,000 audio-text
pairings, to facilitate the audio and text alignments. Our experiments
demonstrate that AVicuna can effectively handle TRD in audio-visual videos and
achieve state-of-the-art performance on various audio-visual video
understanding tasks, particularly in untrimmed videos. We further investigate
the optimal audio-interleaving rate for interleaved audio-visual inputs, which
maximizes performance on the Audio-Visual Event Dense Localization task.