In this paper, we propose the Associative Conversation Model that generates visual information from textual information and uses it for generating sentences in order to utilize visual information in a dialogue system without image input. In research on Neural Machine Translation, there are studies that generate translated sentences using both images and sentences, and these studies show that visual information improves translation performance. However, it is not possible to use sentence generation algorithms using images for the dialogue systems since many text-based dialogue systems only accept text input. Our approach generates (associates) visual information from input text and generates response text using context vector  fusing associative visual information and sentence textual information. A comparative experiment between our proposed model and a model without association showed that our proposed model is generating useful sentences by associating visual information related to sentences. Furthermore, analysis experiment of visual association showed that our proposed model generates (associates) visual information effective for sentence generation. As a model that can extract knowledge from conversations, the encoder-decoder model has been proposed BID12 BID14 . It consists of an encoder that encodes the input information into a context vector and a decoder that generates sentences using the context. BID14 showed that it is possible to extract knowledge and to conduct conversation by learning pairs of dialogues with the model. For example, BID14 reported that when asked who is Skywalker, their conversation model (NCM) responded "he is a hero." NCM has a problem that it is not possible to respond properly to the input texts that require visual information. For example, BID14 reported that when asked how many legs a spider have, NCM responded "three, i think." Further, the image or video may contain more detailed information than texts. Consider, for example, a scene in a news program including a closed caption "one marathon runner won the marathon competition " and showing an image with the marathon runner with the gold medal. Here, in the video, more detailed information such as the gold medal that does not exist directly in the text is presented. We thought that if such detailed visual information could be extracted from the image, more specific and useful texts could be generated, including "gold medals" which can not be obtained with text alone. In recent years, studies have been reported in which translated sentences are generated by adding image features to the context vector encoded by the encoder-decoder model BID1 BID3 BID7 BID8 BID13 . These studies showed that visual information works effectively for generating translation.Meanwhile, visual information is not considered in many text-based dialogue systems, because what is given to the input is only the utterance text. How can the visual information be used without accepting visual information as the input to the dialogue system? Based on the discussion above, we propose an Associative Conversation Model that associates the input text with the visual information and generates the response using both the text and the asso- Figure 1 : Generating a response by visual association. The textual information is used to estimate the corresponding visual information, and a response text is generated using the vector obtained by fusing the textual and visual information. ciated visual information. In our proposed method, we attempted to generate response texts using visual information without inputting images. The contribution of this research is as follows:• We made it possible to generate visual information related to sentence textual information through end-to-end learning of dialogue.• We made it possible to generate sentences using visual information without directly inputting visual information by association. • Our proposed model can generate response texts including useful information compared with a model without association by associating visual information related to input text. Our method is useful for constructing the text-based dialogue systems that automatically extract information from the text and the video data (e.g., TV news) to generate sentences. In a study applying a sentence generation algorithm of translation sentence to a conversation model, there was a problem that it was not possible to respond well to an input text which requires visual information. However, it is not possible to use sentence generation algorithms using images for the dialogue systems since many text-based dialogue systems only accept text input. Based on the discussion above, we propose an Associative Conversation Model that associates the input text with the visual information and generates the response using both the text and the associated visual infor-mation. Comparative experiments with models that do not use association show that association of visual information related to input texts produces response texts that contain valuable information compared to models without association. Analysis of association also showed that our proposed method can generate visual information related to sentence textual information through end-to-end learning of dialogue. Our method is useful for constructing the text-based dialogue systems that automatically extract information from the text and the video data (e.g., TV news) to generate sentences.
Proposal of the sentence generation method based on fusion between textual information and visual information associated with the textual information This work describes a deep learning model for dialogue systems that takes advantage of visual information. This paper proposes a novel dataset for grounded dialog and makes a computational observation that it could help to reason about vision even when performing text-based dialog. Proposes to augment traditional text-based sentence generation/dialogue approaches by incorporating visual information by collecting a bunch of data consisting of both text and associated images or video