Unified Language Representation for Question Answering over Text,
  Tables, and Images
Bowen Yu Cheng Fu Haiyang Yu Fei Huang Yongbin Li
When trying to answer complex questions, people often rely on multiple
sources of information, such as visual, textual, and tabular data. Previous
approaches to this problem have focused on designing input features or model
structure in the multi-modal space, which is inflexible for cross-modal
reasoning or data-efficient training. In this paper, we call for an alternative
paradigm, which transforms the images and tables into unified language
representations, so that we can simplify the task into a simpler textual QA
problem that can be solved using three steps: retrieval, ranking, and
generation, all within a language space. This idea takes advantage of the power
of pre-trained language models and is implemented in a framework called Solar.
Our experimental results show that Solar outperforms all existing methods by
10.6-32.3 pts on two datasets, MultimodalQA and MMCoQA, across ten different
metrics. Additionally, Solar achieves the best performance on the WebQA
leaderboard