Visual Analytics for Efficient Image Exploration and User-Guided Image
  Captioning
Yiran Li Junpeng Wang Prince Aboagye Michael Yeh Yan Zheng Liang Wang Wei Zhang Kwan-Liu Ma
Recent advancements in pre-trained large-scale language-image models have
ushered in a new era of visual comprehension, offering a significant leap
forward. These breakthroughs have proven particularly instrumental in
addressing long-standing challenges that were previously daunting. Leveraging
these innovative techniques, this paper tackles two well-known issues within
the realm of visual analytics: (1) the efficient exploration of large-scale
image datasets and identification of potential data biases within them; (2) the
evaluation of image captions and steering of their generation process. On the
one hand, by visually examining the captions automatically generated from
language-image models for an image dataset, we gain deeper insights into the
semantic underpinnings of the visual contents, unearthing data biases that may
be entrenched within the dataset. On the other hand, by depicting the
association between visual contents and textual captions, we expose the
weaknesses of pre-trained language-image models in their captioning capability
and propose an interactive interface to steer caption generation. The two parts
have been coalesced into a coordinated visual analytics system, fostering
mutual enrichment of visual and textual elements. We validate the effectiveness
of the system with domain practitioners through concrete case studies with
large-scale image datasets.