Data Interpretation is an important part of Quantitative Aptitude exams and requires an individual to answer questions grounded in plots such as bar charts, line graphs, scatter plots, \textit{etc}. Recently, there has been an increasing interest in building models which can perform this task by learning from datasets containing triplets of the form \{plot, question, answer\}. Two such datasets have been proposed in the recent past which contain plots generated from synthetic data with limited (i) $x-y$ axes variables (ii) question templates and (iii) answer vocabulary and hence do not adequately capture the challenges posed by this task. To overcome these limitations of existing datasets, we introduce a new dataset containing $9.7$ million question-answer pairs grounded over $270,000$ plots with three main differentiators. First, the plots in our dataset contain a wide variety of realistic $x$-$y$ variables such as CO2 emission, fertility rate, \textit{etc. } extracted from  real word data sources such as World Bank, government sites, \textit{etc}. Second, the questions in our dataset are more complex as they are based on templates extracted from interesting questions asked by a crowd of workers using a fraction of these plots. Lastly, the answers in our dataset are not restricted to a small vocabulary and a large fraction of the answers seen at test time are not present in the training vocabulary. As a result, existing models for Visual Question Answering which largely use end-to-end models in a multi-class classification framework cannot be used for this task. We establish initial results on this dataset and emphasize the complexity of the task using a multi-staged modular pipeline with various sub-components to (i) extract relevant data from the plot and convert it to a semi-structured table (ii) combine the question with this table and use compositional semantic parsing to arrive at a logical form from which the answer can be derived. We believe that such a modular framework is the best way to go forward as it would enable the research community to independently make progress on all the sub-tasks involved in plot question answering. Data plots such as bar charts, line graphs, scatter plots, etc. provide an efficient way of summarizing numeric information and are frequently encountered in textbooks, research papers, professional reports, newspaper articles, etc. Machine comprehension of these plots with the aim of answering questions grounded in them is an interesting research problem which lies at the intersection of Language and Vision and has widespread applications. For example, such a system could enable financial analysts to use natural language questions to access the information locked in a collection of data plots within financial reports, journals, etc. Such a system could also serve as an important educational assistant for the visually impaired by helping them understand the information summarized in a plot by asking a series of questions.Recently, two datasets , viz., FigureQA BID6 and DVQA BID5 have been released for this task which contain triplets of the form {plots, questions, answers}. These datasets clearly show that despite its apparent similarity to Visual Question Answering (VQA), this task has several additional challenges due to which existing state of the art VQA methods do not perform well on this task. However, both FigureQA and DVQA themselves have some limitations which warrants the creation of more challenging datasets which adequately capture a wider range of Figure 1 shows a sample triplet from FigureQA, DVQA and DIP (our dataset). First, we note that FigureQA and DVQA contain plots created from synthetic data. In particular, note that the label names (either x-axis or y-axis or legend names) in FigureQA and DVQA come from a limited vocabulary (color names and top-1000 nouns from Brown corpus respectively). This clearly reduces the vocabulary that a model needs to deal with. Further, the label names are not really meaningful in the context of the plot leading to unnatural questions. In contrast, the plots in our dataset are based on World Bank Open Data which contains realistic variable names such as mortality rate, crop yield, country names, etc. The values associated with each plot are also realistic with different scales including floating point numbers as opposed to DVQA which contains only integers in a fixed range. Secondly, the questions in FigureQA and DVQA are based on a smaller number of templates (15 and 25). Instead of hand designing a small number of templates, we first show a fraction of the plots to crowdsourced workers and ask them to create natural language questions which can be answered from these plots. We then analyze these questions and extract templates from them which results in a richer question repository with more templates (74) and more complex questions. Lastly, unlike FigureQA and DVQA the answers in our dataset do not come from a small fixed set of vocabulary. For example, the answer to the question shown in FIG0 is 60.49, which is not present in the training data. More specifically, the answer vocabulary for the test data is 248, 878 words of which 187, 688 are not seen in the training data. This is quite natural and expected when the plots and questions are extracted from real world data. In addition to the above differentiators, we also include an extra novel test set which contains plots based on data extracted from Open Government Data as opposed to World Bank Data. This test set contains additional variable and legend names which are not seen in the World Bank Data.Given the large answer vocabulary, it is infeasible to use any of the existing VQA models on our dataset as they largely treat VQA as a multi-class classification problem where the task is to select the right answer from a fixed vocabulary. Even the recent models proposed on DVQA take a similar multi-class classification approach. Further, we believe that given the various sub-tasks involved in this problem it is not prudent to use a single end-to-end system which simply takes the plot and question as input and generates as answer. Instead, as a baseline we propose a modular multi-staged pipeline wherein the first stage in the pipeline extracts (i) data objects in the plot such as bars, lines, etc.(ii ) text objects in the plot such as titles, legend names, x-y axes names, etc. (iii ) the numeric objects in the plot such as tick values, scales, etc. At the end of this stage, the plot is converted to a semi-structured table. We then use a method BID13 which combines the question with this table and uses compositional semantic parsing to arrive at a logical form from which the answer can be derived. The key point here is that the output is neither selected from a fixed vocabulary nor generated using a decoder but it is derived from a logical form. Our experiments using this model suggest that this dataset is indeed very challenging and requires the community to look beyond existing end-to-end models. We introduced the DIP dataset for Data Interpretation over Plots which contains scientific plots created from real world data sources such as World Bank, stock market, etc. Further, the questions in our dataset are based on templates which were manually extracted from realistic questions created by crowd-workers. One of the primary challenges of this dataset is that it has a large vocabulary because of which existing VQA models which treat this as a multi-class classification model cannot be applied. Instead we propose a modular pipeline which first converts the plot to a semi-structured table and then learns to answer questions from this table using compositional semantic parsing. Our experiments suggest that this is a very challenging dataset and requires significant progress on multiple sub-tasks. In particular, we need improved models for reasoning over structured data.Ray Smith. TAB7 presents the detailed statistics of the number of plots present in each of our data splits according to plot type. Note that the plot type k-multi means that the number of lines/bars on each tick is k. In the above mentioned samples, the horizontal bar graph is 3-multi and so on. (e) In how many <plural form of X label>, is the <Y label> of/in <legend label> greater than <N> <units>? (f) What is the ratio of the <Y label> of/in <legend label1> in < i th x tick> to that in < j th x tick>? (g) Is the <Y label> of/in <legend label> in < i th x tick> less than that in < j th x tick> ? 5. Compound : (a) Is the difference between the <Y label> in < i th x tick> and < j th x tick> greater than the difference between any two <plural form of X label> ? (b) What is the difference between the highest and the second highest <Y label> ? (c) Is the sum of the <Y label> in < i th x tick> and < (i + 1) th x tick> greater than the maximum <Y label> across all <plural form of X label> ? (d) What is the difference between the highest and the lowest <Y label> ? (e) In how many <plural form of X label>, is the <Y label> greater than the average <Y label> taken over all <plural form of X label> ? (f) Is the difference between the <Y label> of/in <legend label1> in < i th x tick> and < j th x tick> greater than the difference between the <Y label> of/in <legend label2> in < i th x tick> and < j th x tick> ? (g) What is the difference between the highest and the second highest <Y label> of/in <legend label> ? (h) What is the difference between the highest and the lowest <Y label> of/in <legend label> ? (i) In how many <plural form of X label>, is the <Y label> of/in <legend label> greater than the average <Y label> of/in <legend label> taken over all <plural form of X label> ? (j) Is it the case that in every <singular form of X label>, the sum of the <Y label> of/in <legend label1> and <legend label2> is greater than the <Y label> of/in <legend label3> ? (k) Is the sum of the <Y label> of/in <legend label1> in < i th x tick> and < j th x tick> greater than the maximum <Y label> of/in <legend label2> across all <plural form of X label>? (l) Is it the case that in every <singular form of X label>, the sum of the <Y label> of/in <legend label1> and <legend label2> is greater than the sum of <Y label> of <legend label3> and <Y label> of <legend label4> ? D. QUESTION TYPES TAB8 presents the distribution of the number of questions, categorized by their template type, that are present in each of the data splits.
We created a new dataset for data interpretation over plots and also propose a baseline for the same. The authors propose a pipeline to solve the DIP problem involving learning from datasets containing triplets of the form {plot, question, answer} Proposes an algorithm that can interpret data shown in scientific plots.