Towards Adaptable and Interactive Image Captioning with Data
  Augmentation and Episodic Memory
Aliki Anagnostopoulou Mareike Hartmann Daniel Sonntag
Interactive machine learning (IML) is a beneficial learning paradigm in cases
of limited data availability, as human feedback is incrementally integrated
into the training process. In this paper, we present an IML pipeline for image
captioning which allows us to incrementally adapt a pre-trained image
captioning model to a new data distribution based on user input. In order to
incorporate user input into the model, we explore the use of a combination of
simple data augmentation methods to obtain larger data batches for each newly
annotated data instance and implement continual learning methods to prevent
catastrophic forgetting from repeated updates. For our experiments, we split a
domain-specific image captioning dataset, namely VizWiz, into non-overlapping
parts to simulate an incremental input flow for continually adapting the model
to new data. We find that, while data augmentation worsens results, even when
relatively small amounts of data are available, episodic memory is an effective
strategy to retain knowledge from previously seen clusters.