Weakly-supervised Automated Audio Captioning via text only training
Theodoros Kouzelis Vassilis Katsouros
In recent years, datasets of paired audio and captions have enabled
remarkable success in automatically generating descriptions for audio clips,
namely Automated Audio Captioning (AAC). However, it is labor-intensive and
time-consuming to collect a sufficient number of paired audio and captions.
Motivated by the recent advances in Contrastive Language-Audio Pretraining
(CLAP), we propose a weakly-supervised approach to train an AAC model assuming
only text data and a pre-trained CLAP model, alleviating the need for paired
target data. Our approach leverages the similarity between audio and text
embeddings in CLAP. During training, we learn to reconstruct the text from the
CLAP text embedding, and during inference, we decode using the audio
embeddings. To mitigate the modality gap between the audio and text embeddings
we employ strategies to bridge the gap during training and inference stages. We
evaluate our proposed method on Clotho and AudioCaps datasets demonstrating its
ability to achieve a relative performance of up to ~$83\%$ compared to fully
supervised approaches trained with paired target data.