Improving Image Captioning Descriptiveness by Ranking and LLM-based
  Fusion
Simone Bianco Luigi Celona Marco Donzella Paolo Napoletano
State-of-The-Art (SoTA) image captioning models often rely on the Microsoft
COCO (MS-COCO) dataset for training. This dataset contains annotations provided
by human annotators, who typically produce captions averaging around ten
tokens. However, this constraint presents a challenge in effectively capturing
complex scenes and conveying detailed information. Furthermore, captioning
models tend to exhibit bias towards the ``average'' caption, which captures
only the more general aspects. What would happen if we were able to
automatically generate longer captions, thereby making them more detailed?
Would these captions, evaluated by humans, be more or less representative of
the image content compared to the original MS-COCO captions? In this paper, we
present a novel approach to address previous challenges by showcasing how
captions generated from different SoTA models can be effectively fused,
resulting in richer captions. Our proposed method leverages existing models
from the literature, eliminating the need for additional training. Instead, it
utilizes an image-text based metric to rank the captions generated by SoTA
models for a given image. Subsequently, the top two captions are fused using a
Large Language Model (LLM). Experimental results demonstrate the effectiveness
of our approach, as the captions generated by our model exhibit higher
consistency with human judgment when evaluated on the MS-COCO test set. By
combining the strengths of various SoTA models, our method enhances the quality
and appeal of image captions, bridging the gap between automated systems and
the rich, informative nature of human-generated descriptions. This advance
opens up new possibilities for generating captions that are more suitable for
the training of both vision-language and captioning models.