With a Little Help from your own Past: Prototypical Memory Networks for
  Image Captioning
Manuele Barraco Sara Sarto Marcella Cornia Lorenzo Baraldi Rita Cucchiara
Image captioning, like many tasks involving vision and language, currently
relies on Transformer-based architectures for extracting the semantics in an
image and translating it into linguistically coherent descriptions. Although
successful, the attention operator only considers a weighted summation of
projections of the current input sample, therefore ignoring the relevant
semantic information which can come from the joint observation of other
samples. In this paper, we devise a network which can perform attention over
activations obtained while processing other training samples, through a
prototypical memory model. Our memory models the distribution of past keys and
values through the definition of prototype vectors which are both
discriminative and compact. Experimentally, we assess the performance of the
proposed model on the COCO dataset, in comparison with carefully designed
baselines and state-of-the-art approaches, and by investigating the role of
each of the proposed components. We demonstrate that our proposal can increase
the performance of an encoder-decoder Transformer by 3.7 CIDEr points both when
training in cross-entropy only and when fine-tuning with self-critical sequence
training. Source code and trained models are available at:
https://github.com/aimagelab/PMA-Net.