Attention-Based Methods For Audio Question Answering
Parthasaarathy Sudarsanam Tuomas Virtanen
Audio question answering (AQA) is the task of producing natural language
answers when a system is provided with audio and natural language questions. In
this paper, we propose neural network architectures based on self-attention and
cross-attention for the AQA task. The self-attention layers extract powerful
audio and textual representations. The cross-attention maps audio features that
are relevant to the textual features to produce answers. All our models are
trained on the recently proposed Clotho-AQA dataset for both binary yes/no
questions and single-word answer questions. Our results clearly show
improvement over the reference method reported in the original paper. On the
yes/no binary classification task, our proposed model achieves an accuracy of
68.3% compared to 62.7% in the reference model. For the single-word answers
multiclass classifier, our model produces a top-1 and top-5 accuracy of 57.9%
and 99.8% compared to 54.2% and 93.7% in the reference model respectively. We
further discuss some of the challenges in the Clotho-AQA dataset such as the
presence of the same answer word in multiple tenses, singular and plural forms,
and the presence of specific and generic answers to the same question. We
address these issues and present a revised version of the dataset.