Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in
  Chart Captioning
Kung-Hsiang Huang Mingyang Zhou Hou Pong Chan Yi R. Fung Zhenhailong Wang Lingyu Zhang Shih-Fu Chang Heng Ji
Recent advancements in large vision-language models (LVLMs) have led to
significant progress in generating natural language descriptions for visual
content and thus enhancing various applications. One issue with these powerful
models is that they sometimes produce texts that are factually inconsistent
with the visual input. While there has been some effort to mitigate such
inconsistencies in natural image captioning, the factuality of generated
captions for structured document images, such as charts, has not received as
much scrutiny, posing a potential threat to information reliability in critical
applications. This work delves into the factuality aspect by introducing a
comprehensive typology of factual errors in generated chart captions. A
large-scale human annotation effort provides insight into the error patterns
and frequencies in captions crafted by various chart captioning models,
ultimately forming the foundation of a novel dataset, CHOCOLATE. Our analysis
reveals that even state-of-the-art models, including GPT-4V, frequently produce
captions laced with factual inaccuracies. In response to this challenge, we
establish the new task of Chart Caption Factual Error Correction and introduce
CHARTVE, a model for visual entailment that outperforms proprietary and
open-source LVLMs in evaluating factual consistency. Furthermore, we propose
C2TFEC, an interpretable two-stage framework that excels at correcting factual
errors. This work inaugurates a new domain in factual error correction for
chart captions, presenting a novel evaluation mechanism, and demonstrating an
effective approach to ensuring the factuality of generated chart captions.