Improving Audio Caption Fluency with Automatic Error Correction
Hanxue Zhang Zeyu Xie Xuenan Xu Mengyue Wu Kai Yu
Automated audio captioning (AAC) is an important cross-modality translation
task, aiming at generating descriptions for audio clips. However, captions
generated by previous AAC models have faced ``false-repetition'' errors due to
the training objective. In such scenarios, we propose a new task of AAC error
correction and hope to reduce such errors by post-processing AAC outputs. To
tackle this problem, we use observation-based rules to corrupt captions without
errors, for pseudo grammatically-erroneous sentence generation. One pair of
corrupted and clean sentences can thus be used for training. We train a neural
network-based model on the synthetic error dataset and apply the model to
correct real errors in AAC outputs. Results on two benchmark datasets indicate
that our approach significantly improves fluency while maintaining semantic
information.