Putting Humans in the Image Captioning Loop
Aliki Anagnostopoulou Mareike Hartmann Daniel Sonntag
Image Captioning (IC) models can highly benefit from human feedback in the
training process, especially in cases where data is limited. We present
work-in-progress on adapting an IC system to integrate human feedback, with the
goal to make it easily adaptable to user-specific data. Our approach builds on
a base IC model pre-trained on the MS COCO dataset, which generates captions
for unseen images. The user will then be able to offer feedback on the image
and the generated/predicted caption, which will be augmented to create
additional training instances for the adaptation of the model. The additional
instances are integrated into the model using step-wise updates, and a sparse
memory replay component is used to avoid catastrophic forgetting. We hope that
this approach, while leading to improved results, will also result in
customizable IC models.