SciCap+: A Knowledge Augmented Dataset to Study the Challenges of
  Scientific Figure Captioning
Zhishen Yang Raj Dabre Hideki Tanaka Naoaki Okazaki
In scholarly documents, figures provide a straightforward way of
communicating scientific findings to readers. Automating figure caption
generation helps move model understandings of scientific documents beyond text
and will help authors write informative captions that facilitate communicating
scientific findings. Unlike previous studies, we reframe scientific figure
captioning as a knowledge-augmented image captioning task that models need to
utilize knowledge embedded across modalities for caption generation. To this
end, we extended the large-scale SciCap
dataset~\cite{hsu-etal-2021-scicap-generating} to SciCap+ which includes
mention-paragraphs (paragraphs mentioning figures) and OCR tokens. Then, we
conduct experiments with the M4C-Captioner (a multimodal transformer-based
model with a pointer network) as a baseline for our study. Our results indicate
that mention-paragraphs serves as additional context knowledge, which
significantly boosts the automatic standard image caption evaluation scores
compared to the figure-only baselines. Human evaluations further reveal the
challenges of generating figure captions that are informative to readers. The
code and SciCap+ dataset will be publicly available at
https://github.com/ZhishenYang/scientific_figure_captioning_dataset