DiffAVA: Personalized Text-to-Audio Generation with Visual Alignment
Shentong Mo Jing Shi Yapeng Tian
Text-to-audio (TTA) generation is a recent popular problem that aims to
synthesize general audio given text descriptions. Previous methods utilized
latent diffusion models to learn audio embedding in a latent space with text
embedding as the condition. However, they ignored the synchronization between
audio and visual content in the video, and tended to generate audio mismatching
from video frames. In this work, we propose a novel and personalized
text-to-sound generation approach with visual alignment based on latent
diffusion models, namely DiffAVA, that can simply fine-tune lightweight
visual-text alignment modules with frozen modality-specific encoders to update
visual-aligned text embeddings as the condition. Specifically, our DiffAVA
leverages a multi-head attention transformer to aggregate temporal information
from video features, and a dual multi-modal residual network to fuse temporal
visual representations with text embeddings. Then, a contrastive learning
objective is applied to match visual-aligned text embeddings with audio
features. Experimental results on the AudioCaps dataset demonstrate that the
proposed DiffAVA can achieve competitive performance on visual-aligned
text-to-audio generation.