JaSPICE: Automatic Evaluation Metric Using Predicate-Argument Structures
  for Image Captioning Models
Yuiga Wada Kanta Kaneda Komei Sugiura
Image captioning studies heavily rely on automatic evaluation metrics such as
BLEU and METEOR. However, such n-gram-based metrics have been shown to
correlate poorly with human evaluation, leading to the proposal of alternative
metrics such as SPICE for English; however, no equivalent metrics have been
established for other languages. Therefore, in this study, we propose an
automatic evaluation metric called JaSPICE, which evaluates Japanese captions
based on scene graphs. The proposed method generates a scene graph from
dependencies and the predicate-argument structure, and extends the graph using
synonyms. We conducted experiments employing 10 image captioning models trained
on STAIR Captions and PFN-PIC and constructed the Shichimi dataset, which
contains 103,170 human evaluations. The results showed that our metric
outperformed the baseline metrics for the correlation coefficient with the
human evaluation.