Audio Difference Captioning Utilizing Similarity-Discrepancy
  Disentanglement
Daiki Takeuchi Yasunori Ohishi Daisuke Niizumi Noboru Harada Kunio Kashino
We proposed Audio Difference Captioning (ADC) as a new extension task of
audio captioning for describing the semantic differences between input pairs of
similar but slightly different audio clips. The ADC solves the problem that
conventional audio captioning sometimes generates similar captions for similar
audio clips, failing to describe the difference in content. We also propose a
cross-attention-concentrated transformer encoder to extract differences by
comparing a pair of audio clips and a similarity-discrepancy disentanglement to
emphasize the difference in the latent space. To evaluate the proposed methods,
we built an AudioDiffCaps dataset consisting of pairs of similar but slightly
different audio clips with human-annotated descriptions of their differences.
The experiment with the AudioDiffCaps dataset showed that the proposed methods
solve the ADC task effectively and improve the attention weights to extract the
difference by visualizing them in the transformer encoder.