Enhanced Chart Understanding in Vision and Language Task via Cross-modal
  Pre-training on Plot Table Pairs
Mingyang Zhou Yi R. Fung Long Chen Christopher Thomas Heng Ji Shih-Fu Chang
Building cross-model intelligence that can understand charts and communicate
the salient information hidden behind them is an appealing challenge in the
vision and language(V+L) community. The capability to uncover the underlined
table data of chart figures is a critical key to automatic chart understanding.
We introduce ChartT5, a V+L model that learns how to interpret table
information from chart images via cross-modal pre-training on plot table pairs.
Specifically, we propose two novel pre-training objectives: Masked Header
Prediction (MHP) and Masked Value Prediction (MVP) to facilitate the model with
different skills to interpret the table information. We have conducted
extensive experiments on chart question answering and chart summarization to
verify the effectiveness of the proposed pre-training strategies. In
particular, on the ChartQA benchmark, our ChartT5 outperforms the
state-of-the-art non-pretraining methods by over 8% performance gains.