Text-to-Audio Generation Synchronized with Videos
Shentong Mo Jing Shi Yapeng Tian
In recent times, the focus on text-to-audio (TTA) generation has intensified,
as researchers strive to synthesize audio from textual descriptions. However,
most existing methods, though leveraging latent diffusion models to learn the
correlation between audio and text embeddings, fall short when it comes to
maintaining a seamless synchronization between the produced audio and its
video. This often results in discernible audio-visual mismatches. To bridge
this gap, we introduce a groundbreaking benchmark for Text-to-Audio generation
that aligns with Videos, named T2AV-Bench. This benchmark distinguishes itself
with three novel metrics dedicated to evaluating visual alignment and temporal
consistency. To complement this, we also present a simple yet effective
video-aligned TTA generation model, namely T2AV. Moving beyond traditional
methods, T2AV refines the latent diffusion approach by integrating
visual-aligned text embeddings as its conditional foundation. It employs a
temporal multi-head attention transformer to extract and understand temporal
nuances from video data, a feat amplified by our Audio-Visual ControlNet that
adeptly merges temporal visual representations with text embeddings. Further
enhancing this integration, we weave in a contrastive learning objective,
designed to ensure that the visual-aligned text embeddings resonate closely
with the audio features. Extensive evaluations on the AudioCaps and T2AV-Bench
demonstrate that our T2AV sets a new standard for video-aligned TTA generation
in ensuring visual alignment and temporal consistency.