Improving Cross-modal Alignment with Synthetic Pairs for Text-only Image
  Captioning
Zhiyue Liu Jinyuan Liu Fanrong Ma
Although image captioning models have made significant advancements in recent
years, the majority of them heavily depend on high-quality datasets containing
paired images and texts which are costly to acquire. Previous works leverage
the CLIP's cross-modal association ability for image captioning, relying solely
on textual information under unsupervised settings. However, not only does a
modality gap exist between CLIP text and image features, but a discrepancy also
arises between training and inference due to the unavailability of real-world
images, which hinders the cross-modal alignment in text-only captioning. This
paper proposes a novel method to address these issues by incorporating
synthetic image-text pairs. A pre-trained text-to-image model is deployed to
obtain images that correspond to textual data, and the pseudo features of
generated images are optimized toward the real ones in the CLIP embedding
space. Furthermore, textual information is gathered to represent image
features, resulting in the image features with various semantics and the
bridged modality gap. To unify training and inference, synthetic image features
would serve as the training prefix for the language decoder, while real images
are used for inference. Additionally, salient objects in images are detected as
assistance to enhance the learning of modality alignment. Experimental results
demonstrate that our method obtains the state-of-the-art performance on
benchmark datasets.