Marrying Dialogue Systems with Data Visualization: Interactive Data
  Visualization Generation from Natural Language Conversations
Yuanfeng Song Xuefang Zhao Raymond Chi-Wing Wong
Data visualization (DV) has become the prevailing tool in the market due to
its effectiveness into illustrating insights in vast amounts of data. To lower
the barrier of using DVs, automatic DV tasks, such as natural language question
(NLQ) to visualization translation (formally called text-to-vis), have been
investigated in the research community. However, text-to-vis assumes the NLQ to
be well-organized and expressed in a single sentence. However, in real-world
settings, complex DV is needed through consecutive exchanges between the DV
system and the users. In this paper, we propose a new task named CoVis, short
for Conversational text-to-Visualization, aiming at constructing DVs through a
series of interactions between users and the system. Since it is the task which
has not been studied in the literature, we first build a benchmark dataset
named Dial-NVBench, including dialogue sessions with a sequence of queries from
a user and responses from the system. Then, we propose a multi-modal neural
network named MMCoVisNet to answer these DV-related queries. In particular,
MMCoVisNet first fully understands the dialogue context and determines the
corresponding responses. Then, it uses adaptive decoders to provide the
appropriate replies: (i) a straightforward text decoder is used to produce
general responses, (ii) an SQL-form decoder is applied to synthesize data
querying responses, and (iii) a DV-form decoder tries to construct the
appropriate DVs. We comparatively evaluate MMCoVisNet with other baselines over
our proposed benchmark dataset. Experimental results validate that MMCoVisNet
performs better than existing baselines and achieves a state-of-the-art
performance.